RTD FP7 PROPOSAL EVALUATION

Jo Prieur

Covent Garden, the FP7 Proposal Evaluation Building

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

General Principles

Evaluation by Commission based on peer review by independent experts, with: – Transparency – Equality of Treatment, Impartiality & Fairness – Confidentiality – Efficiency & Speed (???) – Quality/Excellence – Ethical & Security considerations IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Facts & figures

• 16 000 proposals / year in FP6, on average • 5000 evaluators/year in FP6, on average • An expert data base of over 50 000 names • Over 26 000 proposals in FP7 (incl.9000 for ERC) for 2007 for a budget of € 6 billion, in 57 Calls.

• 6700 evaluators appointed for 2007 (close to 9000 with ERC referees) • A dedicated building (Covent Garden) for evaluations, all year round IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Overall evaluation scheme

Proposal Submission Individual reading Consensus Panel (+hearings) Evaluators Criteria Evaluators Criteria Evaluators Criteria answers Follow up by Commission Final ranking list Rejection list Expert involvement

COMMISSION COMMISSION

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Evaluation steps

• Evaluation criteria, scores & thresholds • Selection of evaluators • Commission officials • Proposal eligibility • Evaluator briefing • Individual evaluations • Consensus meetings and reports • Hearings (if applicable) • Panel • Finalisation IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • • • – – –

Evaluation criteria, scores & thresholds

3 major criteria (with funding scheme-dependent sub-criteria) S/T Quality (concept, objectives, workplan) Implementation (individual participants, consortium, resources) Impact (expected impacts listed in WP, dissemination, exploitation) Scores from 0 to 5 for each criteria (not for sub-criteria) Individual thresholds (3) for each criteria Weightings if specified (in general no weighting) Overall threshold (10) higher than the weighted sum of individual thresholds

(«being good enough is not good enough»)

Relevance (FP6 criteria) integrated in S/T quality and impact …but clear irrelevance (“out of scope”) means “non eligible” IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Scoring scale

3 4 5 0 1 2

The proposal fails to address the criterion under examination or cannot be judged due to

missing or incomplete

information

Very poor -

The criterion is addressed in a cursory and unsatisfactory manner.

Poor -

There are serious inherent weaknesses in relation to the criterion in question.

knesses that could not be corrected at the later changing the project)

Fair -

While the proposal broadly addresses the criterion, there are significant weaknesses that would need correcting.

Good -

The proposal addresses the criterion well, although certain improvements are possible.

Excellent -

The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor Half marks may be given …..Full score range should be used!

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Selection of experts/evaluators

• Independent expert database made from calls for individual candidates and to organisations to appoint experts https://cordis.europa.eu/emmfp7/ • Experts selected « ad personam » (not representing their employer, nor country, nor any other entity) • Expertise & range of competences are paramount for the choice of evaluators. These are assessed from data (CV) in the database (and possibly from earlier participation in proposal evaluations) • Balance between countries, types and sizes of organisations (large companies/SMEs, industry / research / academia) and gender aspects are taken into account as long as above main criteria are satisfied.

• Sufficient renewal (about 30% new experts for each call) • Constraints: availability, avoidance of conflicts of interest, number and coverage of proposals IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Role of experts / evaluators

• Experts agree to terms and conditions of an “appointment letter” with clear rules on confidentiality and conflicts of interest. They adhere to a code of conduct.

• An individual will review typically 7 to 10 proposals in 4/5 days (for small CPs, including consensus meetings… “productivity” very much instrument-dependent • Some will participate in panel work • Some will participate in hearings • Names of evaluators are published once a year, per theme, after the evaluations http://cordis.europa.eu/fp7/experts_en.html

• One evaluator is acting as an independent observer: he “evaluates the evaluation”, assure the process is fair, and make a report and recommendations to the Commission IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Role of Commission Officials

• Check eligibility of proposals and put each eligible proposal in a predefined group/category (sub/theme, activity, etc.) • Brief the evaluators and answer relevant questions from them • Oversee the work of the evaluators and ensure a confidential, fair, coherent and consistent evaluation process according to established criteria, rules and procedures.

• Control the information available to the evaluators (proposals, earlier evaluation results for resubmitted proposals, etc.) • Collect/Compile individual evaluation results • Moderate the consensus meetings and control quality of the consensus reports (completeness, correctness, fairness) • Do not attempt to influence the opinion of the evaluators nor express views about any proposal or applicant.

• Organise panel meetings and hearings IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Eligibility Check

• Timely proposal receipt (before closing date and time) • Minimum number of independent partners as requested by the rules of participation, or the call, or the work programme • Proposal completeness (all parts and forms) • Formal submission as requested in the call (e.g. electronic submission only via EPSS) • Any other specific criteria (i.e. no classified information in Security proposals) • Out of scope • Out of budget limits

New for FP7 New for FP7 Difficult cases submitted to an internal eligibility committee

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Evaluation groups

• • • Eligible proposals grouped into categories according to Work Programme (sub-theme, activity, area, etc.) Evaluation group of experts assigned to each category Number of evaluators in each group based on expected workload (number of proposals for the group and expected «productivity» of the experts: typically from 2/ 3 readings a day for a small CP to 1 reading in 2 days for a large CP or IP, including consensus)

Difficult exercise due to uncertainty on the expected number of proposals

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • •

Evaluator briefing by the Commission

Explain what is expected from the evaluators (evaluation guidelines, objectives, procedures, criteria, scoring, thresholds, weightings, timing) Explain the role of the observer and of the rapporteurs Answer questions from evaluators May provide general information about the call such as overall number of proposals, over subscription ratio… or recommendations (typically: be fair but severe, use full score range, be coherent between scores & comments, etc.) Explain specific situations (e.g. handling of sensitive information for Security proposals) IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Individual evaluations

• Number of independent readings: from 3 minimum for a small CP to 5 for a large CP • Evaluators should declare any potential conflict of interest before starting reading • Individual scores & comments, based only on the written proposal (no reading between the lines!) are entered into an IER (Individual Evaluation Report) • No discussion between evaluators at this stage • No contact with any partner of the proposals, no telephone, no computer, all documents locked in the evaluation room in the evening, etc.

• Under the scrutiny of the Commission assisted by the independent observer IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • •

Consensus

For each proposal, all evaluators come together for a consensus meeting chaired by the moderator Meeting aims at an agreed view on the proposal, and comments and scores accepted by all (typical duration: from < 0.5 hour, up to several hours). It is a true discussion/debate, not a mere averaging exercise!

In case of lasting disagreements, additional evaluators (up to 3) can be called in.

The consensus report is written by the rapporteur (one evaluator or the moderator), signed by all evaluators and moderator.

The moderator ensures that the discussion is coherent, complete and fair, that each evaluator can express his views, that conclusions are properly reflected in the consensus report and that conclusions do support the scores.

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • • •

Hearings (Large CP, IP or NoE only)

Hearings take place a few days after the write up of the consensus report, only for proposals above all thresholds Objective is to clarify the proposal (and not to modify or rewrite, or to make a show) Hearing based solely on a set of questions prepared in writing by the evaluators during the consensus meeting.

Hearing conducted by the same evaluators as the initial evaluation (plus extra experts on specific issues if necessary) A hearing report is prepared.

If necessary, the consensus report can be modified (scores & comments) in the light of the hearing results IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • •

Panel

Key function is to ensure overall consistency Composed of a limited number of evaluators of each group and moderated by a Commission official Review and discuss all consensus reports and arrive at a collective consensus on each proposal (harmonisation and standard of scoring between evaluation groups) Write an ESR (Evaluation Summary Report) for each proposal (scores & comments), not necessarily identical to the initial consensus report…but deviations must be justified.

Establish and sign the evaluation report containing all ESR and recommendations on proposal ranking (split equal rating proposals), possible clustering of proposals, grant agreement negotiation, etc.

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Ethical Review

• Triggered by the experts or the Commission • Carried out on the proposal submitted – Only exceptionally will additional information be requested • The ER considers, for example: – ethical aspects and the social impact of the research – respect for the FP7 ethical standards – application of relevant European Directives, conventions etc – whether the consortium is seeking the approval of relevant local ethics committees – balance between the research objectives and the means IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • • • •

Finalisation

Approval of ESR and issuing ESR to coordinators (initial information letter, redress procedure)

New for FP7

Establishment by the Commission of the final list taking into account evaluation results, in consultation with other relevant services of the Commission.

Programme committee information and discussion Establishment of the lists of projects for negotiations and of a reserve list taking into account budgetary constraints.

Notification of rejected proposals to coordinators Preparation & approval of an implementation plan Notification of successful proposals to coordinators and start of negotiations.

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Redress Procedure

• • • • • • • • Included in the rules of participation by the Council Complaints to be submitted within 1 month from the initial information letter (before selection/rejection decisions) Complaints to be based only on eligibility and the evaluation process Calling into question the judgement of appropriately qualified experts is not permitted A redress committee will examine the complaints Redress procedure rarely leads to re-evaluation (4 case out of 700 in 2007) Benefits: coherent and consistent approach to complaints, transparency & equal treatment Difficulty (for the Commission): labor-intensive IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

• • • • • •

Keep in mind (1)

Fail to prepare= Prepare to fail

. Time is of the essence. Start early, do not wait for the call to come out: read important documents, even in draft form, and build up your partnership.

The 2 “bibles” to be read carefully are the work programme and the guide for applicants … Read them carefully… and read the right ones!

Read also how your proposal will be evaluated ftp://ftp.cordis.europa.eu/pub/fp7/docs/fp7-evrules_en.pdf

Understand the structure of the Work Programme (Activities, Areas, Topics) and the “jargon” (Levels 1 & 2, Funding Schemes, etc.) Do not ignore the general parts of the WP: they give you contextual information and references which will help you positioning your project within a broader policy, society and/or strategy context (useful for the “impact” aspects) Build up your partnership: no alibi-partner. All must have a clear function and bring a real added value to your project IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Keep in mind (2)

• • • • • Do not deviate from the required structure of the proposal (the template). It is designed to ensure practicality & commonality, to match the evaluation criteria, and make life of the Commission officers and evaluators easier Do not write too much: over-long proposals are rarely viewed in a positive light… the limits on the number of pages is important. Avoid repeating entire paragraphs.

Do not write too little. Cover all what is required in part A (the forms) and B (the text) of your proposal. Do not assume, but demonstrate. Leave nothing to the imagination of the evaluators: they are not allowed to have any! They are instructed to make their judgments solely on the basis of what they read, nothing else!...

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Keep in mind (3)

• • • – – – Don’t forget your major objective at the submission stage: convince well informed and educated evaluators Who have been selected to make judgements against 3 equal criteria But…who may not be specialists of your particular topic And…who are just human beings: they do not know everything, they get tired by the end of the evaluation week, etc.

Use simple words. Avoid specialist jargon. Be concise and accurate. Avoid acronyms or explain them Don’t waste time with coloured graphics or images. Evaluators have only black & white copies to look at. Make sure that illustrations and graphics, if any, are meaningful to the evaluators.

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

Keep in mind(4)

• • • • • All evaluation criteria are important and not only scientific & technological excellence. Impact has become more important in FP7.

Perform an internal evaluation before submitting your proposal (quality assurance), and therefore allow enough time for an iteration.

Submit, re-submit, re-re-submit …only the latest submitted version stays in the system and will be evaluated (if eligible) Register in the database to become an evaluator: work …but It’s real hard also an occasion to meet new faces, network, and understand what makes a good proposal. It will help you for your future proposal preparation.

For hearings, answer the questions, all questions, nothing but the questions.

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels

A typical evaluation room

IGLO Training Session Sept. 7, 2009 Jo Prieur, CLORA, 8 Avenue des Arts, 1210 Brussels