Transcript Dr. Sebastiano Fumero

The evaluation process in the
7th Framework programme
for Research and Technological
Development
Chisinau, November 6th, 2012
Dr Sebastiano FUMERO
Head of Unit “FP7 support”
European Commission Research Executive Agency
Policy
REA - Research
Executive Agency
Overview
 How to apply
 The evaluation process:
basic facts and figures
 Role of Commission/REA staff
 Key issues:
•
Eligibility check
•
Expert selection
•
Conflicts of interest
•
The criteria
•
The observer
•
Redress
FP7: how to apply
Participant
Annual
portal
Work
Programme
Calls for
proposals
Year N
Guide for
Applicants
Call X
Budget, deadline,
OJ ref., legal
documents
Funding
Scheme Y
Links to
SEP
SEP
Electronic
proposal
submission
system
NCP
The Participant Portal
(PP)
The Participant Portal
(PP)
The Participant Portal
(PP)
Submission
To consider prior to submission
 Rules on submission and evaluation
• The common reference for FP7
• Consistency vs flexibility!
 Guide for applicants (annexes 1 and 2)
• The common reference for FP7
 The work programme
• The topics and criteria against which the proposals will be
judged (all criteria are important – consider sub-criteria think as an evaluator)
7
Submission
Drafting the Proposal
• Respect page limits specified in guide for applicants
• Ensure you meet the minimum eligibility requirements
• Excellent science is a condition but not enough. Consider also:
- impact, dissemination and IPS
- consider project implementation and management (role of
coordinator is essential)
• Be precise, less is sometime more…
• Impartial view…ask your colleagues, friend to read it before…
• Start with SEP asap – a missed deadline implies proposal is not
admissible
• administrative data (part A forms) should be consistent with
info in part B
8
Evaluation process
Submission
Full Proposal
Proposal
forms
Individual
evaluation
Evaluators
Criteria
Consensus
Panel
Evaluators
Evaluators
Criteria
Criteria
Finalisation
Final ranking
list
Rejection list
Proposals in
suggested
priority order
Eligibility
COMMISSION
COMMISSION
Experts' role
Evaluation process
Proposal
Eligibility
Individual Evaluation Remote or in Bruxelles
Consensus
Thresholds
Quick Information Letter
Panel review
Hearings
For large projects
(optional)
Commission ranking
Applicants are informed
of the Commission
decision
Negotiation
Commission rejection
decision
Commission funding
decision
Evaluation of proposals
Basic facts and figures
 Funding decisions are based on peer review of research
proposals
•
Peer review can also “add value” to projects
 High quality evaluators are at the core of the system
 Over 118.000 experts registered for FP7 in the old
database
 Over 15.000 experts registered in the new database
(Expert Area in the Participant Portal)
 Over 8 300 independent experts engaged in 2011
 Approx. 27 000 proposals evaluated in 2011
Evaluation of proposals
Evaluation Process: basic principles
Evaluation of proposals
Three References
RULES
 Rules on submission and evaluation
Submission
&
Evaluation
• The common reference for FP7
• Consistency vs flexibility!
 Guide for applicants (annexes 1 and 2)
• The common reference for FP7
Guide for
Applicants
Call X
Funding
Scheme Y
 The work programme
• The topics and criteria against which the proposals
will be judged
Annual
Work
Programme
Evaluation of proposals
Some basic
misconceptions
clarified
 The EU’s peer review
system is not a political
process
•
Lobbying has no influence
 Quality of the proposal is
the sole criterion for
success
•
However, “quality” involves a
number of factors
Evaluation of proposals
Role of Commission/REA staff
 Check the eligibility of proposals
 Oversee work of experts
 Conduct briefings
 Moderate discussions
 Organise the panel and its work
 Ensure coherence and consistency
 May advise on:
•
Background on previously supported or on-going projects
•
Relevant supplementary information (directives, regulations, policies, etc.)
 (Can even act as experts!)
Evaluation of proposals
Eligibility checks
 Receipt of proposal before deadline
•
Firm deadlines (SEP)
 Minimum number of eligible, independent
partners
•
As set out in work programme and the call
 Completeness of proposal
•
Presence of all requested forms
•
and readable, accessible and printable
 "In scope" vs "Out of scope"
 Others
Evaluation of proposals
Expert selection
 Based on:
•
A high level of expertise
•
An appropriate range of competences
 If the above conditions can be satisfied, then also:
•
Balance academic/industrial
•
Gender
•
Geography
•
Rotation
 But also, of course constrained by:
•
Availability
•
Avoidance of conflicts of interest
•
Uncertainty over number and exact coverage of proposals
 Not an easy process…!!!
FP7 Expert
from this area
Number of experts having supported the EC
in FP7 by country
Country
Number of Experts
(participation)
Greece
EL
646
Romania
RO
365
Hungary
HU
311
Bulgaria
BG
165
Slovenia
SI
163
Slovakia
SK
95
Croatia
HR
76
Serbia
RS
62
Macedonia
MK
8
Bosnia and Herzegovina
BA
5
Moldova
MD
3
Number of experts registered in the EMPP by country
Country
Migrated
Valid
Bosnia and Herzegovina
BA
8
8
Bulgaria
BG
204
162
Croatia
HU
115
89
Greece
HR
914
687
Hungary
HU
297
204
Macedonia
MK
25
21
Moldova
MD
16
12
Romania
RO
464
361
Serbia
RS
101
78
Slovakia
SK
121
94
Slovenia
SI
166
123
Expert selection
Conflicts of interest (1)
 More clarity in FP7
 Types of COI set out in appointment letter
 Disqualifying COI
•
Involved in preparation of proposal
•
Stands to benefit directly
•
Close family relationship
•
Director/trustee/partner
•
Employee (but, see exception…)
•
•
Member of advisory group
Any other situation that compromises impartiality
 Potential COI
•
Employed in last 3 years
•
•
Involved in research collaboration in previous 3 years
Any other situation that casts doubt… or that could reasonably appear to do so…
Expert selection
Conflicts of interest (2)
 Experts with a “disqualifying” COI cannot evaluate
•
•
•
Neither in consensus group considering “problematic” proposal
Nor in final panel
One exception… if:



•
… then the Commission/REA might allow expert to participate in a
panel review

•
The expert is employed in same organisation, but different
department/lab/institute (e.g. CNRS)
The constituent bodies operate with a high degree of autonomy
Justified by the limited pool of qualified experts
Should abstain if the specific proposal is discussed
Exceptionally (very rare!!!), might participate in consensus group
 Experts with a “potential” COI
•
Need to consider circumstances of case
The evaluation criteria
1. S/T quality
(in relation to the
topics addressed
by the call)
Sound concept, and
quality of objectives
2. Implementation
• Appropriateness of the
management structure
and procedures
• Quality and relevant
experience of the
individual participants
3. Impact
Contribution, at the
European and / or
international level, to the
expected impacts listed
in the work programme
under the relevant topic
/ activity
The evaluation criteria
Some exceptions
 Marie-Curie schemes for
training and mobility of
researchers
•
Include, e.g. quality of training
programme, suitability of host
institution, etc.
 European Research Council
(ERC) grants
• Scientific quality only criterion
Excellence!!!
Proposal scoring
 Each criterion is scored 0-5
• Half-scores allowed
• Whole range should be considered
• Scores must pass thresholds if a proposal is to be
considered for funding
 Thresholds apply to individual criteria:
• Default threshold is 3
 … and to the total score
• Higher than the sum of the individual thresholds
• Default threshold is 10
 Can vary from call to call!
Proposal scoring
Interpretation of the scores
0 - The proposal fails to address the criterion under
examination or cannot be judged due to missing or incomplete
information
1 - Poor. The criterion is addressed in an inadequate manner, or
there are serious inherent weaknesses.
2 - Fair. While the proposal broadly addresses the criterion,
there are significant weaknesses.
3 - Good. The proposal addresses the criterion well, although
improvements would be necessary.
4 - Very Good. The proposal addresses the criterion very well,
although certain improvements are still possible.
5 - Excellent. The proposal successfully addresses all relevant
aspects of the criterion in question. Any shortcomings are minor.
Evaluation process
Individual reading
The experts evaluators first carry out individual readings
(often done remotely)
The experts:
Evaluate the proposal individually (without discussing with the
other evaluators)
Check whether the proposal is “in scope”
second check after the one done by the EC
Complete an Individual Evaluation Report (IER)
giving scores and comments on all criteria
•
Scores should be in line with comments
Evaluation process
May be remote
Proposal 1
Proposal 1
Proposal 1
IER
Individual
Evaluation
Report
Expert A
Consensus
meeting
Consensus:
Scores &
comments
IER
Expert B
IER
Expert C
HEARING
(optional)
One proposal can be evaluated by more than 3 experts
CR
Consensus
Report
Evaluation process
Consensus
 Build on the basis of the
individual assessments of
all the evaluators
 Usually involves a
discussion
 Moderated by a
Commission/ REA
representative
 Agreement on consensus
scores and comments for
each of the criteria
 One expert acts as
rapporteur
Evaluation process
Consensus reports – key points
 The rapporteur is responsible for drafting the
consensus report (CR)
•
Includes consensus marks and comments
 The quality of the CR is paramount
 The aim is
•
•
A clear assessment of the proposal, with justification
Clear feedback on weaknesses and strengths
 To be avoided
•
•
Comments that do not correspond with the scores
Recommendations in view of resubmission
A proof reader might be appointed for quality control
Evaluation process
Hearings
 Co-ordinators whose
proposals have passed
the thresholds are invited
to Brussels
 Intended to clarify any
points raised by the
experts in advance
 Not an occasion to
“improve” the proposal
 Not an occasion for a
multi-media show!
Evaluation process
The final Panel Review
 Key function is to ensure consistency
 Final marks and comments for each proposal
•
•
•
Evaluation Summary reports (ESR)
New scores (if necessary)… carefully justified
Clear guidance for contract negotiation
 Split proposals with identical consensus scores
•
Approach is spelled out in WP and GFA
 Resolve cases where a minority view was recorded
in CR
 [Exceptionally] recommendations for combining
 List of proposals for priority order
Information to proposers
“Initial information” to applicants
Sending of ESR
 The Commission/REA does not change the ESR,
except if necessary to:
•
•
Improve readability
[Exceptionally] To remove factual errors or inappropriate
comments that may have escaped earlier proof-reading
 The scores are never changed
 The ESR is sent to the proposal co-ordinator – no
commitments at this stage regarding funding
 This is the public face of the evaluation!
Appeal
Redress?
 In the past, complaints arrived haphazardly
•
•
•
Handled at different levels
No systematic treatment
No common record
 The redress procedure introduced for FP7 does
not give a new right of appeal…… but it ensures
a consistent and coherent approach to
complaints
•
•
Establishes “due process”
Uphold principles of transparency and equal treatment
Appeal
Redress: Principles and guidelines
 Redress will not “stop the train”
•
Non-contentious proposals negotiated and selected as
normal
 Complaints must relate to shortcomings in the
handling of proposal evaluation
•
Before a Commission decision has been made
 The procedure will not call into question the
judgement of appropriately qualified experts
Monitoring
Independent
observers
 Provide assurance that the
process is fair
•
•
And can provide constructive
advice
Not experts in the scientific
area concerned
 Their reports are made
available to the
Programme Committee
 IOs are convened annually
to a Round Table
•
What are the common issues?
Evaluation process
Commission/REA follow-up
 Evaluation summary reports sent to applicants
 Draw up final ranking lists
 Information to the Programme Committee
 Contract negotiation
 Formal consultation of Programme Committee (when
required)
 Commission decisions
 Survey of evaluators
 Independent Observers’ reports
Survey
Expert questionnaire
 For every call, experts receive a message on returning
home
 Invited to complete an on-line survey
•
•
•
•
•
•
Personal profile
Evaluation process
Evaluation criteria
Opinion on the task and the other evaluators
Logistics
Comments and recommendations
 Early results sent to call co-ordinator after one month
 Full analysis at end of the year
Survey
96% of the respondents found the quality of
the evaluation overall 'satisfactory' to
'excellent'
Survey
91% found the EU evaluation process similar or
better than national or international schemes
Links
EU research: http://ec.europa.eu/research/
7th Framework Programme:
http://ec.europa.eu/research/fp7/
Information on research activity and projects:
http://cordis.europa.eu/
Questions? Contact the Research Enquiry Service
http://ec.europa.eu/research/index.cfm?pg=enquiries
Policy
REA - Research
Executive Agency
Thank you for your attention!
Dr. Sebastiano FUMERO
Head of Unit “FP7 Support”
European Commission
Research Executive Agency
Tel: +32-2-296 96 88
[email protected]
Policy
REA - Research
Executive Agency