Transcript Evaluation and benchmarking of Humanities research in Europe

Impact assessment in the Humanities:
problems and prospects
Carl Dolan
Vienna, 15 December 2008
[email protected]
Presentation Roadmap
Introductions
HERA study
Basic data issues, bibliometrics etc
A framework for evaluation and
benchmarking in the humanities
Feasible benchmarking: a case
study
Introductions
• My role - disclaimer #1
• Humanities in the European Research
Area (HERA)
– Network of 18 humanities funding agencies
(www.heranet.info)
• Address common challenges e.g.
– ex-ante peer review
– joint programming and double jeopardy
– benchmarking research quality
HERA report on the evaluation and
benchmarking of humanities research
• HERA work-package (12 mths)
• Coordinated by AHRC
• Survey of evaluation practices in
agencies (questionnaire)
• Survey of ex-post evaluation systems in
Europe (desktop research)
• Review of developments in bibliometrics
and peer review
• 2 workshops in London (Mar 06/Jan
07)
HERA report on the evaluation and
benchmarking of humanities research
• Disclaimer #2 – feasibility study,
not new research
• Response to immediate policy
concerns – devise an informed,
common position
HERA report on the evaluation and
benchmarking of humanities research
• Short answer – credible benchmarking
system not feasible (in short term)
• Basic input, activity & output data just
isn’t there (or is patchy)
• Four main data sources
–
–
–
–
National Statistics Agencies, OECD, Eurostat
Universities
Funding agencies
Commercial bibliometric databases
International availability of data (OECD)
• HERD for humanities 1981-2002
– 11 of 30 countries had no data
– only 2-11 per year, lots of gaps
– Germany, Denmark 14 yrs; Czech 4;
France 0
• R&D personnel
– No data before 1995
– 7 of 30 had patchy data
• Research postgraduate ………..
Country
Australia
Austria
Belgium
Belgium (Flemish Community)
Canada
Czech Republic
Denmark
Finland
France
Germany
Greece
Hungary
Iceland
Ireland
Italy
Japan
Korea
Luxembourg
Mexico
Netherlands
New Zealand
Norway
Poland
Portugal
Slovak Republic
Spain
Sweden
Switzerland
Turkey
United Kingdom
United States
Field of study
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
Humanities (ISC 22)
ISCED level 6 data = PhD equivalent
1998
376
0
1999
440
228
61
454
33
199
0
1,645
334
0
74
58
466
52
46
197
909
1,635
322
0
92
568
920
548
0
0
240
0
30
0
568
988
530
0
54
241
1
67
0
764
185
179
258
0
5,188
50
702
180
146
352
0
5,419
2000
436
188
127
530
57
0
225
1,389
1,963
152
0
36
96
543
1,116
581
0
63
0
0
57
0
146
29
673
166
130
266
1,287
5,456
2001
443
207
148
90
0
215
1,472
1,793
187
0
62
135
534
1,190
11
61
222
1
79
0
30
770
194
132
237
1,691
5,200
Universities
• Collect large amounts of data on outputs, staff
numbers, grant income etc
• Data not easily accessible in all countries
• Avoid imposing additional administrative burdens on
universities where possible
• Few European countries where there is systematic
collection and collation of university data across all
fields
–
–
–
–
–
UK (RAE)
Netherlands (VSNU)
Slovenia
Poland
Norway (to a lesser extent)
Funding agencies
• Survey reveals common practice (EoA
reporting forms) and large amounts of output
data
• But…
–
–
–
–
Data difficult to manipulate
Under-reporting of project outcomes
No significant agencies in some countries
Representative? - Relative share of total humanities
research output varies and is typically small
• Trend toward creation of research information
systems/ CV repositories (DFG pilot, RCUK in
pipeline)
Commercial bibliometric databases
• Poor coverage of output due to:
– National/regional focus
– US bias
– Language issues
• 70% English in AHCI
– Journals only
• 61% references to monographs in Hist & Phil Sci
– Specialist journals only (non-academic audiences
excluded)
– Different citation practices /windows
– Team oriented v. single scholar
– Differences in academic culture?
HISTORY in Thomson journal databases
1800
1600
Numbers of journal articles
1400
1200
1000
800
600
400
200
0
1981 1982 1983 1984 1985 1986 1987 1988 1989 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004
CANADA
CZECH REPUBLIC
FRANCE
GERMANY
UK
USA
Response to criticisms
• Most criticisms of ISI AHCI
• Expand coverage: combine databases (Google
scholar, Scopus, CSA illumina)
• Possible to ‘mine’ AHCI for more ‘raw citations’
• Evidence of growing ‘journal culture’
• Evidence that AHCI has good coverage of
‘important international journals in fields with
a dominant international research frontier’
(Nederhof)
• US bias not clear (in terms of performance)
Database developments
• Avoid citation databases
• Use book and journal weights
• Construct weights through peer-review
and consultation
• E.g. Norwegian Ministry
– National Library and ISI give 90% coverage
– Common documentation system
– Classified according to publication type and
publication channel
Weighting of publications in Norwegian
Model
Channels at
normal level
Channels at
high level
Table 1. The weighting of publications
Articles
ISSN-titles
in 1
3
Articles
ISBN-titles
in 0,7
1
Books
titles)
(ISBN- 5
8
Could Norwegian model be expanded?
•
2 necessary elements
i. Common documentation system
ii. Agreement on weights
i.
SI (SICRIS), NL (DAREnet), FI
(KOTA), CA (Common CV), UK?
-
Link and standardise?
ii. European Reference Index (ERIH)
Problems with this approach
• Need for constant revision and
consultation
• Risk of perverse incentives and
stifling emerging research
• Measuring impact of channels, not
publications
A framework for evaluation in the
humanities: common principles
• Importance of better data
collection
• Importance of quantifiable
evidence (objectivity &
transparency)
• Importance of peer judgement
• Proxies for peer judgement are
available
A framework for evaluation in the
humanities: common principles
• Importance of disciplinary variation
(within humanities)
• Assessment at disciplinary level –
data not fine-grained enough
• A holistic approach –suite of
indicators
• Applies to other disciplines too –
no A&H exceptionalism
A feasible framework for international
benchmarking? Case study
• Dicennial Assessment of PhD
programmes in US (NRC)
–
–
–
–
Every 10 years since 1960s
200 US institutions surveyed
All fields inc. humanities
Ranked list based on quality assessment
• Evaluation methodology revised
– Pre-2007: Reputational method
– Now: ‘implicit method’
Implicit method
Obtain reputation measures for a sample of programs
in each field (anchor study)
Use multiple regression to construct a model of predictors of
reputation ratings
Construct separate indicators for each field using weights
obtained from questionnaire survey and statistical models
Use model to impute reputation measures for non-sampled
programs with appropriate error bands (not simple ranking)
Implicit method
• Constructs indicators by weighting the
following categories (faculty quality)
 Number of publications per faculty member
 Number of citations per faculty member
 Receipt of extramural grants for research
 Involvement in interdisciplinary work
 Racial/ethnic diversity of program faculty
 Gender diversity of program faculty
 Reception by peers of a faculty member’s
work, as measured by honors and awards
A feasible framework for international
benchmarking? Implicit method
 Peer review at its heart (anchoring study +
consultation on weightings)
 Guided by appropriate quantitative data
 Sensitive to disciplinary variation
 Holistic -broad range of quantitative data
considered
 Effectively same method for all disciplines
 Avoids problems of bias and non-replicable
nature of peer review
 Feasible to apply to large quantities of data
Thank you for your attention