Transcript Departmental h-index : Case study for Greek universities

Evaluating Greek Departments of
Computer Science/Engineering
using Bibliometric Indices
Dimitris Katsaros1,2
Vassilis Matsoukas1
Yannis Manolopoulos1
1Informatics
Department, Aristotle University, Thessaloniki
2Department of Computer & Communications Eng, University of Thessaly, Volos
12th Panhellenic Conference on Informatics
Samos Island, August 28-30, 2008
Scientometrics: the Science of Measuring
Science
• Careers in science are not scientific; they
depend on:
• luck
• social connections
• the ability to impress influential people and
referees
• the foresight to join the right lab at the right
time
• the foresight to associate oneself with
prestigious people and prestigious projects
• Such systems waste scientific talent and
produce resentment
2
Scientometrics: the Science of Measuring
Science
• Promotion strictly according to scientific
merit would revolutionize scientific career
• Scientific production: the basis for any
measurement of scientific merit
• Scientific production consists of:
• published articles, and
• their impact
• Scientometrics: the science of
measuring the “quality” of science
3
Why Measuring Science?
• Numerical indices for quantification of published
research output are being increasingly used by:
• employers for hiring personnel
• promotion panels – promotions, tenure
• funding agencies – “Funding does not
regenerate funding. But reputation does.”
• Australia: Research Quality Framework (RQF)
• UK: Research Assessment Exercise (RAE)
• Attempts to quantify the quality of science are
always fraught with difficulty, but it is
worthwhile …
4
Early Approaches: the Impact Factor
• Eugene Garfield (Science, 1972) described the
Impact Factor (IF) for journals:
Impact factor for Journal X, 2007
►A = # citations in all ISI articles during 2007 to
papers published in X during 2005–2006
►B = # of articles published in X during 2005–2006
►Impact Factor = A/B
• The IF is computed from data gathered by
the Institute for Scientific Information (ISI),
which publishes the Science Citation Index
5
Impact Factor Heavily Criticized …
• Few articles make the difference:
• Philip Campbell – Editor-in-Chief of the journal Nature
– concerned about IF’s crudeness (ESEP, 2008):
The value of Nature’s impact factor for 2004 was 32.2
When he analyzed the citations of individual Nature
papers over the relevant period (i.e., citations in 2004
of papers published in 2002 to 2003), he found that
89% of the impact factor was generated by just 25% of
the papers!
6
Impact Factor Heavily Criticized …
• Why papers from a two-year period & citations
from a single year:
• John Ewing concerned about IF’s “parameters”
(NOTICES OF THE AMS, 2006):
Looking at citations for only two years after
publication may produce faulty results
In some fields (e.g., mathematics) citations frequently
follow several years after publication
Why two years?
And why choose citations from journals published in a
single year?
Both are somewhat arbitrary choices, without any
obvious justification
7
The Hirsch h-index
• Jorge Hirsch (PNAS, 2005) defined the h-index:
• An author scores h if h of their N papers each have at
least h citations, with the remaining (N– h) papers
each having fewer than h citations
• Quantifies both the actual scientific
productivity and the apparent scientific
impact of a scientist
• Resists to the power-laws followed by the
evaluation metrics based on simple arithmetics
8
h-index graphical illustration
The intersection of the 45° line with the curve gives h
9
h-index example
• A scientist
with 5 articles
• When ranked,
have 6,4,4,2,1
citations
A Ferrers graph representation
The h-index is
equal to the
length of
the side of the
Durfee square
10
h-index’s shortcomings
• is bounded by the total number of publications:
hÉvariste Galois= 2, hAlbert Einstein=5
• does not consider the context of citations:
• citations are often made simply to flesh-out an
introduction
• citations made in a negative context
• citations made to fraudulent or retracted work
• does not account for confounding factors
• practice of "gratuitous authorship"
• the favorable citation bias associated with review
articles
11
h-index’s shortcomings (2)
• is affected by limitations in citation data bases
• is a natural number and thus lacks
discriminatory power
• does not account for the age of the articles and
the age of citations
• does not account for the number of authors of
a paper
• …and many more
12
The g-index by Egghe
• The h-index de-emphasizes singular successful
publications in favor of sustained productivity. But, it
may do so too strongly!
• Two scientists may have the
same h-index, say, h = 30, but
one has 20 papers that have
been cited more than 1000
times and the other has none
• g-index (Scientometrics, 2006):
the (unique) largest number
such that the top g articles
received (together) at least g2
citations
13
Age-decaying approaches to h-index
• The h-index favors senior scientists
• Contemporary h-index (Scientometrics, 2007)
• A researcher has contemporary h-index hc if
• hc of his Np articles have Sc(i)≥hc
• the rest Np-hc articles have Sc(i)≤hc
• Sc(i)=γ * (Y(now) - Y(i) + 1)-δ |C(i)|
• Usually: γ=4 and δ=1
• The Age-Weighted Citation Rate (AWCR) and AW-index
• AWCR measures the average number of citations to an entire
body of work, adjusted for the age of each individual paper
• An old article gradually loses its “value”
• Show how “active” a researcher is
14
Accounting for multi-authorship
• Individual h-index
• It divides the standard h-index by the average number
of authors in the articles that contribute to the h-index,
in order to reduce the effects of co-authorship
• PoP Individual h-index
• instead of dividing the total h-index, it first normalizes
the number of citations for each paper by dividing the
number of citations by the number of authors for that
paper, then calculates the h-index of the normalized
citation counts
15
More extensions to the h-index …
• Can be extended to measure the performance of:
• Research groups
• Universities/institutions
• Countries
• The notion of successive h-indexes
•
•
•
•
•
Scientometrics, 70(1), 2007: their definition
Scientometrics, 73(1), 2007: for countries
Scientometrics, 75(1), 2008: for institutions
JASIST, 59(8), 2008:
for groups of authors
JASIST, to appear, 2008: mathematical foundation
16
Departmental h-index: the Greek Case
• the h-index of a specific department equals h if
• h of his/her Np faculty members have a value of h as
h-index, and
• the rest (Np−h) faculty members have no more than h
value as h-index
• Inspired by the current debate on Greek
universities evaluation
• Confirm excellence
• Discover patterns and exceptions
17
Data Collection & Processing
• Included 17 departments hosting 552 staff
members of all ranks
• Excluded recently founded departments since
they do not have an adequate number of staff
members
• Paid special attention to avoid name ambiguities
• Use of the tool Publish or Perish
18
Results for the Dept of Informatics (AUTH)
19
Depts. of Computer Science/Engineering
20
The “toppers”
• NTUA, UAthens and UPatras stand high
• UCrete
• is ranked 1st in (almost) all examined bibliometric
indices, even though it is not the most productive
(ranked 8th) due to its medium size (ranked 11th)
• is not very popular among the prospective students
(ranked 14th), which might be attributed to its
distance from the Greek mainland
21
Productivity and Impact Keep Pace
• UCrete and AUEB
show an outstanding
relative quality in
comparison to their
production
• UPatras (Dept of
Elect & Comp Eng)
and UThrace are on
the opposite end
22
h-index & hc-index Keep Pace
AUEB,
UAegean and
UPiraeus
demonstrate a
deviate from the
rule
Due to a recent
successful
recruitment
policy?
23
More Profs  Higher h-index
UCrete and UThessaly
are positive exceptions
to the rule
UThrace and
UMacedonia are
negative exceptions
24
Scientific excellence and popularity
(among the prospective students) keep pace
UCrete and AUTH
(Dept. of Informatics)
seem under-preferred
UThrace, UMacedonia
and AUTH (Dept. of
Elect & Comp Eng)
are over-preferred
25
Summary and Contributions
• Ranking of research institutions by
bibliometric methods: popular tool used by
• policy makers
• public media
• scientific world
• Departmental h-index: the Greek case
• Marked the “best” department
• Recorded some generic patterns
• Discovered “outliers” to these patterns
26