Transcript Research performace evaluation in the Czech Republic

Research performance evaluation
in the Czech Republic:
current methodology and expected
development
Prof. Zdenek Nemecek
Charles University, Prague
Brief history and legal frame
•
The distribution of the government research support is governed by
the law 130/2002 and its consecutive amendments
•
Evaluation of the research entities started in 2004 without any
connection with financing
•
The amendment from 2005 states that the distribution of the support
for the long-term conceptual development of research entities (SCD)
among the support providers (ministries, Academy) should follow the
evaluation of the results
•
Methodics 2008, 2010, 2013
•
The providers are obliged to follow this distribution for their entities
unless they have their own evaluation system
R&D budget
Government sources only, EU funds not included
Total
Grant agencies,
international
cooperation, etc.
SCD
Evaluation Methodics
Keep in mind that:
1 - Methodics is dedicated for division of money among the research entities
2 – it is not supposed to be used on the level of research teams or individuals
3 - it is not intended for identification of excellence
•
Each research entity provides the list of their results to RIV
•
The results are divided into categories – journal paper, proceedings,
patent, etc.
•
The results are further divided according topics – physics, social
sciences, mathematics, chemistry, etc.
•
Methodics assigns a predefined number of points to each result
•
The money that a particular entity receives would be directly
proportional to the sum of the points gained by its results
Topical groups
The number of points gained by a particular topical group is normalized to represent a
given part of the points available (3 000 000)
Medical & Health Sci.
15%
11%
Biological Sci.
12%
17%
Chemical Sci.
Physics
Mathematics
15%
5%
5%
5%
15%
Earth Sci.
Agri. Sci.
Technical Sci. & IT
Social Sci. & Humanities
Categories of results
The numbers of points assigned to publication and application results are normalized to
represent a given part of the points available
3 281 983 points
Journals (WoS)
Journals (SCOPUS, ERIH)
National journals
2% 3%
1%
4%
1%
4%
Books
82.5%
Proceedings
3%
9%
Patents
Utility models
5%
61%
7%
Prototypes
Certified methods
Softw are
Others
17.5%
Evolution of the Methodics
•
The first Methodics 2008 provided a reasonable evaluation of research
entities
•
However, the production of “results for points” started immediately
•
The only category of results not affected by this trend were papers in
good journals (WoS)
•
The consecutive Methodis (2010) introduced several limitations:
- percentage of applied results
- division of results into topics (mathematics, social sciences, etc.) with
predefined number of points
These limitations brought a partial improvement but the main problem –
equal number of points for all results in a given category - remains
Evaluation Methodics 2013
•
Three parts - Mainstays
•
Each mainstay has its own allocation of points
•
Mainstay I – old system for papers in journals indexed in WoS or
Scopus, books and papers in the Czech journals come through peer
review
•
Mainstay II - peer review evaluation of a limited number of results
•
Mainstay III – new system of evaluation of applications, the results are
not weighted and the points are assigned to research entities.
Mainstay weight
•
Mainstay (Pilíř) I – publication results
•
Mainstay (Pilíř) II – NEW – excellent results
•
Mainstay (Pilíř) III – applications
17.5%
10.0%
Pilíř I
Pilíř II
Pilíř III
72.5%
Mainstay I
This mainstay is adopted and adapted from the previous Methodics 2008 and 2010
SHVa,bSci and Humanities
Social
SHVc Sci
Social
Technika aSci
informatika
Technical
and IT
ZemědělstvíSci
Agriculture
Vědy oScizemi
Earth
Matematika
Mathematics
Fyzika
Physics
Chemie
Chemistry
Biologie
Biology
Lékařství
Medicine
Points
WoS
SCOPUS
Jimp
Jimp
Jimp
Jimp
Jimp
Jimp
Jimp
Jimp
Jimp
Jimp
Jsc
Jsc
Jsc
Jsc
Jsc
Jsc
(10 – 305)
ERIH
Domestic Books
Jneimp
Jneimp
(10 – 30)
Jrec
4
BC
BC
BC
BC
BC
BC
Proceedings
D
D
D
D
D
(4 – 120) (8 – 60)
82.5% of points is distributed according to this mainstay
Mainstay II.
• Each research unit submits one excellent result per CZK 10
mil. for evaluation
• The submission is accompanied with reasoning why the
result is excellent
• The result is evaluated by the Expert panel of a given topical
group and 20% of the best is chosen (category A)
• 10% (60 000)points is allocated to this mainstay
• These points will be distributed to the research entities in
accord with the number of results in category A
• Additional 2000 points will be allocated to the entity that
received an ERC grant in course of past 5 years
• Note that this mainstay will be applied from 2015
Estimated number of results
submitted to Mainstay II
celkem:999
999 výsledků
Total:
results
os1
os2
51
110
os3
28 32
133
159
os4
os5
os6
68
149
151
37
81
os7
os8
os9
od10
os11
Mainstay III.
• 105 000 points will be allocated according to this Mainstay
• In the first step, a given number of points will be assigned to
patents, novel races or cultivars
• In the next step, the rest of points will be distributed in accord
with the volume of money gained by a particular entity either
as grants of applied research or by research contracts with
private enterprises
∑Total = α∑Grants + β∑Contracted research
(α = 0.9; β = 0.1)
Present status of M 2013
• Data for Mainstays I and III are collected
• The software tools for the data processing are almost ready
• The members of Expert panels are nominated
• Peer review of books and Czech journals will start in course of June
• The excellent results will be collected in near future
• Since the process according to M 2013 will be applied only on the data
from the last year, it will influence only about 20% of the budget
• Our modeling has shown that no dramatic changes in the money
distribution can be expected