Transcript Jonathan Adams Presentation

THE EVOLVING USE OF DATA IN UNIVERSITY
RESEARCH ASSESSMENT AND MANAGEMENT
History and practice in research assessment
JONATHAN ADAMS, Director, Research & Development
OPEN UNIVERSITY, MARCH 2013
Researchers publish and build on prior
knowledge
(data from Leydesdorff and Wouters)
2
Article records contain rich associated
metadata
Dispersal in freshwater invertebrates
Bilton, D T; Freeland, J R; Okamura, B
ANNUAL REVIEW OF ECOLOGY AND SYSTEMATICS, Volume 32, Pages 159-181. Published: 2001
Times Cited: 265
Citation links and citation impact
Cited References: 147
Movement between discrete habitat patches can present significant challenges to organisms. Freshwater invertebrates achieve
dispersal using a variety of mechanisms that can be broadly categorized as active or passive, and which have important
consequences for processes of colonization, gene flow, and evolutionary divergence. Apart from flight in adult freshwater insects,
active dispersal appears relatively uncommon. Passive dispersal may occur through transport by animal vectors or wind, often
involving a specific desiccation-resistant stage in the life cycle. Dispersal in freshwater taxa is difficult to study directly, and rare but
biologically significant dispersal events may remain undetected. Increased use of molecular markers has provided considerable
insight into the frequency of dispersal in freshwater invertebrates, particularly for groups such as crustaceans and bryozoans that
disperse passively through the transport of desiccation-resistant propagules. The establishment of propagule banks in sediment
Topical keywords
promotes dispersal in time and may be particularly important for passive dispersers by allowing temporal escape from unfavorable
conditions.
KeyWords: ADULT AQUATIC INSECTS; MITOCHONDRIAL-DNA VARIATION; MARKED CADDISFLY LARVAE; DAPHNIA-PULEX
COMPLEX; POPULATION-STRUCTURE; GENE FLOW; NORTH-AMERICAN; EGG BANK; LIMNOPORUS-CANALICULATUS;
MICROSATELLITE ANALYSIS
Collaborating
organisations
Addresses:
[ 1 ] Univ Plymouth, Benth Ecol Res Grp, Dept Biol Sci, Plymouth PL4 8AA, Devon, England
[ 2 ] Open Univ, Dept Biol Sci, Milton Keynes MK7 6AA, Bucks, England
[ 3 ] Univ Reading, Sch Anim & Microbial Sci, Reading RG6 6AJ, Berks, England
Web of Science Categories: Ecology; Evolutionary Biology
Category with cognate literature
3
We index citations because rates vary by field
and average counts grow over time
Biochemistry &
Molecular
Biology
50
40
Nanoscience &
Nanotechnology
30
Chemistry Organic
20
Physics Condensed
Matter
10
Engineering Mechanical
Average citations to papers published in that year
Evolutionary
Biology
0
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Data and analysis: Evidence Thomson Reuters
4
We then find bibliometric impact and peer
review are coherent across institutions
UoA18 Chemistry
Average citation impact for university 1996-2000
1.8
1.6
1.4
Grade 5*
Grade 5
1.2
Grade 4
Grade 3a
1
Grade 3b
0.8
Spearman r = 0.57, P<0.001
0.6
Ratio mapped/NSI = 1.93
0.4
0
1
2
3
4
5
6
Relative citation impact of articles submitted for RAE2001
5
Where did all this come from?
• 1955
– Eugene Garfield’s Science paper on “Citation Indexes for Science”
• 1963
– Science Citation Index (ISI >Thomson >Thomson Reuters)
• 1972
– U.S. National Science Foundation initiates Science Indicators (later
Science & Engineering Indicators), including publication and citation
data
• 1980s
– Uptake of science indicators in Europe; research by SPRU, CWTS,
Hungarian Academy, as well as ISI
• 1992
– Advisory Board for the Research Councils works with ISI on
National Science Indicators to benchmark UK
• 2004
– Elsevier’s Scopus and Google Scholar are launched
6
Where did research evaluation come from?
• 1960
– “The white heat of the technological revolution”, Harold Wilson
• 1970
– “For the scientists, the party is over”, Shirley Williams
• 1980
– UGC/ABRC consensus on selectivity
– 1986, Research Selectivity Exercise
– 1989, Research Assessment Exercise
• 1990
– Evolution of research management and administration
– 1992-2008, RAE - the standard model, evolving grades
– 2014, Research Excellence Framework
7
Research policy and management is about
‘more, better research’
What we
want to
know
Research
quality
Research
black box
8
Output data have underpinned quantitative
research evaluation
What we
want to
know
Research
quality
Research
black box
What we
have to use
O
U
T
P
U
T
S
Journals and
proceedings
Citations
9
You can now use comprehensive research
management information
What evaluators want to know
What research users want to know
Research
quality
Research
scholarships
IDEAS
proposals,
applications
and
partnerships
Charitable
awards
Research
grants
Innovation
funds
Industrial
contracts
What evaluation
needs to use
I
N
P
U
T
S
Research
black box
Numbers –
of researchers,
facilities,
collaboration
Skilled
employment
Trained
people
O
U
T
P
U
T
S
Reports and
grey literature
Journals,
books &
proceedings
O
U
T
C
O
M
E
S
Social policy
change
Citations
Patents
Citation and
address
links
Licences and
spin outs
Deals and
revenue
These data, added to peer review, create
a modern ‘gold standard’
Data and analysis: Evidence Thomson Reuters
10
They are proxy indicators, not metrics
We use multiple ‘bearings’ to assess our uncertainty
11
Responses to evaluation
Research trajectory changed from mid-1980s
UK citation impact
Relative impact of UK research publications
5 yr moving av'ge
1.5
1.4
1.3
1.2
Arrows indicate
RAE years, e.g.
2001 and 2008
1.1
1981
1985
1989
1993
1997
2001
2005
2009
12
Responses to evaluation
Improvement has been pervasive
Average normalised impact (world average = 1.0)
Note that bibliometric indicators are
coherent across RAE peer review grades
16%
1.2
12%
1
17%
0.8
0.6
1991
1992
1993
1994
1995
Grade 4
1996
Grade 3A
1997
Grade 3B
1998
1999
2000
13
Responses to evaluation
Behavioural games - Goodhart’s Law
RAE1996
Science
Outputs
Engineering
%
Outputs
%
Social sciences
Outputs
%
Humanities and arts
Outputs
%
Books and chapters
5,013
5.8
2,405
8.1
16,185
35.1
22,635
44.4
Conference proceedings
2,657
3.1
9,117
30.8
3,202
6.9
2,133
4.2
77,037
89.8
16,951
57.3
22,575
49.0
15,135
29.7
1,104
1.3
1,122
3.8
4,154
9.0
11,128
21.8
1,953
2.5
1,438
5.4
12,972
28.6
25,217
46.5
751
0.9
3,944
14.9
857
1.9
1,619
3.0
76,182
95.8
20,657
78.1
29,449
65.0
17,074
31.5
618
0.8
408
1.5
2,008
4.4
10,345
19.1
Books and chapters
1,048
1.2
216
1.2
12,632
19.0
21,579
47.6
Conference proceedings
2,164
2.5
326
1.8
614
0.9
897
2.0
80,203
93.8
17,451
95.4
50,163
75.5
14,543
32.1
2,125
2.5
301
1.6
3,018
4.5
8,287
18.3
Journal articles
Other
RAE2001
Books and chapters
Conference proceedings
Journal articles
Other
RAE2008
Journal articles
Other
14
The problem with simplistic indicators
• Research activity is complex and very skewed
– Most research evaluation reports averages
– Tables focus on single indicators
– Ranking is even worse
• Average impact can be very misleading
– Research Council studies reveal error of interpretation
– In skewed data, median much smaller than average
– Lots of papers are not cited
– The interesting bit is about how much is really, really cited lots
• So we we prefer Impact Profiles®, Research Footprints® and
bubble diagrams
15
In the ‘changing geography’, China appears
still to lag Europe on research impact ...
UK
1.5
Citation impact relative to world average
Germany
1.25
USA
1
France
0.75
China
0.5
0.25
1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010
Data and analysis: Evidence Thomson Reuters
16
... but an Impact Profile® reveals that China is
already producing excellent research
40%
China 1999-2008 - 499,854 papers
Percentage of output 1999-2008
UK 1999-2008 - 778,936 papers
30%
20%
10%
0%
uncited
RBI > 0 < 0.125 ≥ 0.125 < 0.25
≥ 0.25 < 0.5
≥ 0.5 < 1
≥1< 2
≥2< 4
≥4< 8
≥8
Data and analysis: Evidence Thomson Reuters
17
UK background and ‘golden triangle’
UK higher education sector, all research fields - 306661 papers
UK 'golden triangle', all research fields - 87157 papers
25
This is the small but critical
excess of really highly cited
research output
Percentage of output 2002 - 2006
20
15
10
5
0
uncited
RBI > 0 < 0.125 RBI ≥ 0.125 < RBI ≥ 0.25 < 0.5 RBI ≥ 0.5 < 1
0.25
RBI ≥ 1 < 2
RBI ≥ 2 < 4
RBI ≥ 4 < 8
RBI ≥ 8
18
Evaluation rests on impact as a proxy for
performance, but there is no unique ‘impact’
Biochemistry & Molecular
Biology
Research Footprint®
scales nciF to maximum
value on each axis
EMBL
Oncology
Cell Biology
LMB
MSKCC
Salk
Immunology
Developmental Biology
Scripps
Genetics & Heredity
Data and analysis: Evidence Thomson Reuters
19
Information from multiple cross-comparisons
National Centre for Science and Technology Evaluation, CHINA
Identify principal organizations
publishing research about clean
vehicles in China and USA
Citation impact
China
4.0
3.0
Chinese Academy of
Sciences
Citation impact
Nankai University
US
Zhejiang University
2.0
Shanghai Jiao Tong
University
Tsinghua University
Fudan University
1.0
Central South
University HIT
Xiamen University
Stanford University
4.0
MIT
University of Science
and Technology of
China
US DoE
3.0
scale = 100 papers
UC Berkeley
0.0
0%
10%
20%
30%
40%
2.0
Percentage of papers in top 10%
University of
Maryland
University of Texas
GM
Data & analysis: Thomson Reuters (Evidence)
Argonne National
Laboratory
UC Davis
1.0
University of
Michigan
scale = 100 papers
0.0
0%
10%
20%
30%
40%
Percentage of papers in top 10%
Data & analysis: Thomson Reuters (Evidence)
20
Combine evaluation approaches to address
multiple objectives
• Very few research programmes have a single objective
• Very few scientists agree on how best to evaluate outcomes!
• Some principles:
– Evaluation as part of planning
– Compare like with like, respect diversity
– Recognise merit objectives
• Capacity building, engagement with economy, social benefit
– Recognise research priorities
• Timeliness, pervasiveness, excellence
• Exploitability, applicability, training
– Gather evidence and use quantitative indicators
– Make use of experience and expert judgment
• Risk of conservatism, need for challenge mechanisms
21
Predictive and
prescriptive analytics
Descriptive analytics
Degree of intelligence
How can we use ‘research analytics’ in
evaluation and management?
Optimization
“What’s the best that can happen?”
Predictive modelling,
forecasting
“What will happen next?”
Randomized Testing
“What happens if we try this?”
Statistical models
“Why is this happening?”
Alerts
“What actions are needed?”
Query, drill down
“What exactly is the problem?”
Ad hoc reports
“How many, how often, where?”
Standard Reports
“What happened?”
22
THE EVOLVING USE OF DATA IN UNIVERSITY
RESEARCH ASSESSMENT AND MANAGEMENT
History and practice in research assessment
JONATHAN ADAMS, Director, Research & Development
OPEN UNIVERSITY, MARCH 2013