Transcript Slide 1

Australia’s Research Assessment
Arthur Sale
Emeritus Professor of Computer Science
University of Tasmania
[email protected]
http://eprints.utas.edu.au/8695
History : RQF  ERA
Australia had embarked on planning a national
assessment of the research of its 39
universities – called the Research Quality
Framework (RQF)
In November 2007, the Federal Government
changed as the result of an election.
The incoming Government had promised a
lower cost, simpler evaluation scheme – the
ERA, announced February 2008.
UK Background - RAE
The UK has long operated a Research
Assessment Exercise (RAE) which used
extensive panel assessment, occurred at sixyear intervals, and which determined the
research funding of universities for six years.
The very last RAE was completed in 2008. The
UK government announced a change of policy
prior to that round of assessment.
UK Background - REF
The UK will replace the RAE by a mostly metricbased process, the Research Excellence
Framework (REF).
‘Assessment will combine quantitative indicators - including
bibliometric indicators wherever these are appropriate - and
expert review. Which of these elements are employed, and
the balance between them, will vary as appropriate to each
subject. The economic and social impact of research, as well
as impact upon public policy, will also contribute to the overall
assessment of quality.’
Ironic Fun!
RAE  REF
RQF  ERA
Excellence in Research for Australia
The RQF was based solidly on expert assessment
of discipline groups like the RAE: expensive
ERA uses the experience of the UK’s RAE but
does not copy it, and is biased towards
metrics, with concessions to panel assessment
where essential.
ERA is managed by the Australian Research
Council (ARC) and reports to the Minister for
Innovation, Industry, Science & Research.
Rolling assessment
In another major break with the RAE/REF, the
ERA assessment will be implemented in a
rolling stream two discipline clusters at a time,
rather than all at once in a snapshot year.
Also the Minister has stated that assessment will
not lead to direct financial outcomes for
universities, though this remains to be tested.
Instead it will create data that could lead to
‘league tables’.
Background
Scholarly research is divided into eight discipline
clusters (see next slide).
The groups are defined by the first four digits of
the six digit Australian and New Zealand
Standard Research Classification (ANZSRC).
All research outputs are classified by this
scheme and have been for many years (under
an older version). All Australian repositories
use this scheme (not LC or Dewey).
Discipline clusters
1
2
3
4
5
Physical, Chemical and Earth Sciences (PCE) – 2010
Humanities and Creative Arts (HCA) – 2010
Engineering and Environmental Sciences (EE) – 2011?
Social, Behavioural and Economic Sciences (SBE)
Mathematics, Information and Communication Sciences
(MIC) – 2012?
6
Biological Sciences and Biotechnology (BSB)
7
Biomedical and Clinical Research (BCR) – 2013?
8
Public and Allied Health and Health Services (PAHHS)
ANZSRC (sample)
Preface
Fields of Research
DIVISION 01 MATHEMATICAL SCIENCES
GROUP 0101 PURE MATHEMATICS
GROUP 0102 APPLIED MATHEMATICS
GROUP 0103 NUMERICAL AND
COMPUTATIONAL MATHEMATICS
GROUP 0104 STATISTICS
GROUP 0105 MATHEMATICAL PHYSICS
GROUP 0199 OTHER MATH SCIENCES
DIVISION 02 PHYSICAL SCIENCES
GROUP 0201 ASTRONOMICAL AND SPACE
SCIENCES
GROUP 0202 ATOMIC, MOLECULAR,
NUCLEAR, PARTICLE & PLASMA PHYSICS
GROUP 0203 CLASSICAL PHYSICS
GROUP 0204 CONDENSED MATTER PHYSICS
GROUP 0205 OPTICAL PHYSICS
GROUP 0206 QUANTUM PHYSICS
GROUP 0299 OTHER PHYSICAL SCIENCES
DIVISION 03 CHEMICAL SCIENCES
GROUP 0301 ANALYTICAL CHEMISTRY
GROUP 0302 INORGANIC CHEMISTRY
GROUP 0303 MACROMOLECULAR AND
MATERIALS CHEMISTRY
GROUP 0304 MEDICINAL AND
BIOMOLECULAR CHEMISTRY
GROUP 0305 ORGANIC CHEMISTRY
GROUP 0306 PHYSICAL CHEMISTRY
GROUP 0307 THEORETICAL AND
COMPUTATIONAL CHEMISTRY
GROUP 0399 OTHER CHEMICAL SCIENCEs
DIVISION 04 EARTH SCIENCES
GROUP 0401 ATMOSPHERIC SCIENCES
GROUP 0402 GEOCHEMISTRY
GROUP 0403 GEOLOGY
GROUP 0404 GEOPHYSICS
GROUP 0405 OCEANOGRAPHY
GROUP 0406 PHYSICAL GEOGRAPHY AND
ENVIRONMENTAL GEOSCIENCE
GROUP 0501 ECOLOGICAL APPLICATIONS
Rolling implementation
The first two clusters to be assessed are
– PCE = Physical, Chemical and Earth Sciences
– HCA = Humanities and the Creative Arts (for
example literature, history, architecture, drama
and music)
July 2009 will see a trial evaluation, with the
‘real ERA’ to commence in 2010. It is expected
that two clusters will be assessed per year
giving a four year cycle of evaluation.
2009 Trial
• PCE and HCA are extremes. PCE will be
evaluated mainly by bibliometrics, whereas
HCA won’t use bibliometrics at all but will use
panel evaluation.
• Both will use applied research metrics and
staffing data. This will provide a basic for
future evaluations which will probably be
mixed and more complex.
PCE Submission
Each university will provide an XML document
describing its publication, grant and application
record for the cluster. This will include a
citation of each publication and its Scopus ID.
The ARC will look up each publication and
compile metrics for the university’s cluster (the
SEER system).
The panel will review the metrics and argue
around the margins to arrive at an assessment.
Important Note
Evaluations are not by a research group, but by
all the university’s research output in that
discipline cluster.
Universities must also report their total FTE staff
count in that cluster, and the researcher count
(which includes adjunct and honorary staff).
These are also taken into account in assessing
quality. Research by adjunct staff is also
counted.
Databases
After a tender process, Scopus was selected as the
journal citation data supplier for PCE.
ARC has also created a public tier list of 21,000
journals, ranked by the various professional societies.
The four tiers are
– A*
top 5% (top international standard)
–A
next 15% (excellent, but not in the top tier)
–B
next 30% (good, perhaps local, often PhD outlets)
–C
remaining 50% (the rest)
These will also be used to create metrics.
Ranked Tier List (sample)
18134
5270
3554
4383
5277
10943
1084
4839
1141
1142
1142
1143
1143
1144
1144
7701
3718
8253
7702
18235
20079
1861
31493
10178
21712
12186
30605
12187
33794
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
B
Software Testing Verification & Reliability
Soil & Tillage Research
Soil Dynamics and Earthquake Engineering
Soil Mechanics and Foundation Engineering
Soil Use and Management
Sojourn
Solar Physics
Soldering & Surface Mount Technology
Solid State Communications
Solid State Ionics
Solid State Ionics
Solid State Nuclear Magnetic Resonance
Solid State Nuclear Magnetic Resonance
Solid State Phenomena
Solid State Phenomena
Soudobe Dejiny
Sound and Vibration
South African Archaeological Bulletin
South African Historical Journal
South African Journal of Economics
South African Journal of Education
South African Journal of Geology
South African Journal of Italian Studies
South African Journal of Musicology
South Asia Research
South Atlantic Review
South Carolina Historical Magazine
South Central Review
South Dakota Law Review
0960-0833
0167-1987
0267-7261
0038-0741
0266-0032
0217-9520
0038-0938
0954-0911
0038-1098
0167-2738
0167-2738
0926-2040
0926-2040
1012-0394
1012-0394
1541-0161
0038-1969
0258-2473
0038-2280
1012-0750
0262-7280
0038-2868
0803 COMPUTER SOFTWARE
0503 SOIL SCIENCES
0905 CIVIL ENGINEERING
0905 CIVIL ENGINEERING
0503 SOIL SCIENCES
gen ss/hum
MULTIDISCIPLINARY - SOCIAL SCIENCES/HUMANIT
0201 ASTRONOMICAL AND SPACE SCIENCES
0914 RESOURCES ENGINEERING AND EXTRACTIVE ME
0206 QUANTUM PHYSICS
0206 QUANTUM PHYSICS
0306 PHYSICAL CHEMISTRY (INCL. STRUCTURAL)
0202 ATOMIC, MOLECULAR, NUCLEAR, PARTICLE AND
0206 QUANTUM PHYSICS
0206 QUANTUM PHYSICS
0912 MATERIALS ENGINEERING
2103 HISTORICAL STUDIES
0913 MECHANICAL ENGINEERING
2101 ARCHAEOLOGY
2103 HISTORICAL STUDIES
14 ECONOMICS
13 EDUCATION
0403 GEOLOGY
2005 LITERARY STUDIES
1904 PERFORMING ARTS AND CREATIVE WRITING
gen ss/hum
MULTIDISCIPLINARY - SOCIAL SCIENCES/HUMANIT
2005 LITERARY STUDIES
2103 HISTORICAL STUDIES
2005 LITERARY STUDIES
1801 LAW
HCA Submission
Like PCE, the university will provide an XML
submission, but each publication or piece of
research output will be linked (via URL) to a
document in the university’s repository.
The panel will evaluate the research outputs on
SEER from face evidence and look at a
nominal 20% of the documents.
This will form part of the HCA assessment.
System to Evaluate the Excellence of Research
The ARC’s SEER software provides the central
glue to make this all work.
It will automatically extract data from the XML
submissions and databases and create
summary metric data for review panels.
Expert reviewers will access the university
repositories (eg HCA) through SEER which
(a) will anonymize them, and
(b) has permission to look at restricted documents.
Sampled periods
It is expected that for the 2010 ERA evaluations the census date
and the dates for the reference periods will move forward one
year compared to the trial. That is, the staff census date
would be 31 March 2009, and the reference periods would be
as follows:
Research outputs - 1 January 2003–31 December 2008 (6 years)
Citation analysis - 1 January 2003–1 March 2010 (6 years)
Research income - 1 January 2006–31 December 2008 (3 years)
Applied measures - 1 January 2006–31 December 2008 (3 years)
Summary
• ERA is planned to be low-cost and rolling.
• It is adapted to clusters, and within them to
disciplines.
• Extensive consultation is taking place
• It will reshape Australian research
• I am sure that Australia is willing to share its
experiences with any other country.
• Read the documentation, watch the space...
By now you probably deserve a picture of me.
I am really sorry not to have been with you.
Best wishes, Arthur
Questions?
© Copyright 2009 Arthur Sale
All rights reserved
Arthur Sale asserts the right to be
recognized as author of this work
Contact: [email protected]
Links
• ERA (Aus)
• ANZSRC
• REF (UK)