Transcript Experiences of a Marie Curie Expert Evaluator

Experiences of a Marie Curie
Expert Evaluator
Dr Sara Benetti
Environmental Sciences Research
Institute, University of Ulster
• Vice-chair for FP7 Marie Curie
IEF/IIF/IOF fellowship evaluation
(2013)
• Expert evaluator for FP7 Marie Curie
IEF/IIF/IOF fellowship evaluation
(2011)
• Since 2011: expert evaluator for
European Union FP7 programme
“New and renewable sources of
energy, Energy efficiency &
innovation”
IEF: Intra-European Fellowships;
IIF: International Incoming Fellowships;
IOF: International Outgoing Fellowships;
CAR: Career Restart.
Panel:
•
•
•
•
Project officer/s (EU person/s);
Chair;
Vice-chairs (normally 1 to 8-15 evaluators);
Expert evaluators (lots and from all fields
within the panel topic!).
Marie Curie fellowships: evaluation process (1)
Candidates submit proposals to specific panel
CHE, ECO, ENG, ENV, LIF, MAT, PHY, SOC
Evaluators to declare any
conflict of interest and
select proposals (at least
100) to evaluate based on
the abstract
"1" denotes that the proposal is exactly in the expert's field of expertise;
"2" denotes that the proposal is in the expert's broader field of expertise;
"3" denotes that the proposal is not exactly in the expert's field of expertise s/he
could evaluate it if necessary;
"No" "No expertise" (by default this option is selected in all proposals).
Marie Curie fellowships: evaluation process (2)
REMOTE EVALUATION
Depending on range of expertise, evaluators are
selected and assigned up to 20 proposals to
evaluate in a relatively short period (3-4 weeks)
•
•
•
•
•
IAR
Each proposal has 3 evaluators;
Evaluators will not all be expert in the specific field;
They do not have contact with each others;
They are not allowed to contact the applicant/s.
Proposals are evaluated strictly in relation to the evaluation
criteria;
• If proposal longer than allowed word count, anything over
the word count will NOT be read or evaluated.
Marie Curie fellowships: evaluation process (2)
REMOTE EVALUATION:
scoring
Marie Curie fellowships: evaluation process (2)
REMOTE EVALUATION:
criteria
IEF/IOF
Overall threshold of 70%
Marie Curie fellowships: evaluation process (2)
REMOTE EVALUATION:
criteria
Marie Curie fellowships: evaluation process (2)
REMOTE EVALUATION:
criteria
Marie Curie fellowships: evaluation process (2)
OUTREACH ACTIVITIES
Marie Curie fellowships: evaluation process (3)
Rapporteur to write
CONSENSUS REPORT
Draft CR
The rapporteur is one of the 3 evaluators of a
proposal. He/she has the responsibility to:
• Collate a draft CR taking into account all of the 3
evaluators’ comments.
• Lead the consensus meeting (normally in
Brussels).
• Write up the final CR and get it approved by
other evaluators and proof-read by his/her vicechair before final submission.
Marie Curie fellowships: evaluation process (4)
CONSENSUS MEETING
Remotely or in Brussels, consensus meeting among
3 evaluators takes place.
• The 3 evaluators discuss the contents of the 3 IARs and draft CR;
• Comments in agreement are normally included in CR;
• Conflicting evaluations are discussed in more detail until (if)
agreement among evaluators is reached;
• Marks for each criterion are then decided on the basis of the
evaluations.
2nd draft of CR
Marie Curie fellowships: evaluation process (5)
If agreement evaluators is reached…
• 2nd draft of CR is checked and approved by all evaluators;
• Rapporteur’s vice-chair proof reads, checks for
inconsistencies and match of comments to mark;
• If other evaluators do not approve the report, this process
is reiterated until an agreement is reached on the final
CR.
Final CR submitted to EC
Marie Curie fellowships: evaluation process (5b)
If agreement among evaluators is not reached…
• Second (third/fourth!) meeting may be
scheduled to allow more time for discussion;
• Fourth evaluator may be selected, he/she will
draft an independent IAR and join the following
consensus meeting/s;
• A moderator may be brought in (vice-chair;
chair; project officer.
• A consensus must be reached!
Several iterations of CR until
consensus is reached
Marie Curie fellowships: evaluation process (6)
All proposals in panel are ranked
• Those with an overall mark below threshold are out.
• Those with any mark below the threshold for specific
criteria are out.
• Funding is allocated from highest scoring proposal
going down, depending on funds (this changes from
one year to the next).
Not a rule of thumb but any proposal
below 85% is unlikely to get funded!
EU Marie Curie proposals
Tips for success:
1. Good science;
2. Well explained and at the right level (evaluators may not be
overly familiar with the science but they are scientists and they
will have very limited time to read your proposal);
3. Do your homework: check criteria/requirements/etc. and stick
to them;
4. Make sure you address every single evaluation criterion;
5. You must ‘prove’ everything! It is not enough that the host
institution is ‘well-known’ for…how can you tell?
6. Outreach ‘points’ are easy to get if you make a little effort.
EU Marie Curie funding: the way forward
1.
Name change (politics?);
2.
Criteria will be reduced in
number and simplified (to
avoid repetition as much as
possible);
3.
It is possible more work will
be done remotely (even
harder for evaluators to
discuss pros and cons of
each proposal!);
4.
Stick to the tips for success;
5.
Level of funding may
increase from one year to
the next.
Call for Marie Skłodowska-Curie Individual Fellowships (IF)
H2020-MSCA-IF-2014 out in March 2014