Transcript Diapositiva 1

The Evaluation Process, Tips from an Evaluator’s Point of
View
Leonardo Piccinetti
EFB
OUTLINE
•
•
•
•
•
Evaluation principles
The experts, who are they ?
Role of Commission staff
FP7 evaluation process
FP7 Evaluation Criteria
Basic principles
• Excellence. Projects selected for funding must demonstrate a high
quality in the context of the topics and criteria set out in the calls.
• Transparency. Funding decisions are based on clearly
described rules and procedures, and applicants should
receive adequate feedback on the outcome of the
evaluation of their proposals.
• Impartiality. All proposals submitted to a call
are treated equally. They are evaluated
impartially on their merits, irrespective of
their origin or the identity of the applicants.
Basic principles
• Confidentiality. All proposals and related data, knowledge and documents
communicated to the Commission are treated in confidence.
•
Efficiency and speed. Evaluation, award and grant preparation
should be as rapid as possible, commensurate with maintaining
the quality of the evaluation, and respecting the legal framework.
• Ethical and security considerations: Any proposal
which contravenes fundamental ethical principles, or
which fails to comply with the relevant security
procedures may be excluded at any time from the
process of evaluation, selection and award
The experts, who are they ?
• The Commission draws on a wide pool of evaluators (database) in all
scientific fields
– c. 50,000 in FP6
– Experts/evaluators from ICPC are very welcome!
• Calls for “candidates”
– Call for applications from individuals; and from institutions
– Applications via CORDIS (database of experts)
• A mass-emailing of FP6 experts was sent
– A simple tick-box will ensure registration for FP7
• Commission invites individuals on a call-by-call basis
– Not self-selection!
• Expertise, and experience are paramount
– Geography, gender and “rotation” also considered
Independent experts
• Expert evaluators are at the heart of the FP7 system
• Expert provides independent, impartial and objective
advice to the Commission
represents neither the employer, nor the country!
• Significant funding decisions will be
made on the basis of expert advice
• The integrity of the process is crucial
– Experts have to read the Code of Conduct
annexed to the appointment letter…and
follow it!
Independent experts
• Experts agree to terms and conditions of an “appointment
letter”
• Typically, an individual will review 6-8 proposals
“remotely”….
• …then spend a couple of days in Brussels
• Some will participate in “hearings” with
the consortia
• Travel and subsistence reimbursed
– Plus €450 honorarium per day
• Experts sign confidentiality and conflict of
interest declaration
• Names published after the evaluations
Actors Confidentiality
– The content of proposals, or the evaluation results, can’t be
discussed with anyone
• The sole exception: in the presence of the EC moderator with experts
who are evaluating the same proposal in a consensus meeting group
or final panel
– Is not possible to distribute any documents
related to the evaluation of a proposal, or
take any documents from the evaluation
building
• Note: The Commission publishes names annually,
but as a group – no link between expert and
proposal
Conflicts of interest (2)
• Types of COI set out in appointment letter
– Check the exact wording!
• Disqualifying COI
–
–
–
–
–
–
–
Involved in preparation of proposal
Stands to benefit directly
Close family relationship
Director/trustee/partner
Employee of a partner in a proposal
Member of Advisory Group
Any other situation that compromises impartiality
• Potential COI
– Employed within the last 3 years by a partner in a
proposal
– Involved in research collaboration with proposers in
the previous 3 years
– Any other situation that casts doubt…or that could
reasonably appear to do so…
Role of Commission staff
• Check the eligibility of the proposals
• Oversee work of experts
• Moderate discussions
• Organise the panel and its work
• Ensure coherence and consistency
Evaluation
•
•
•
•
Peer-Review System
Two-stage evaluation procedure
Remote evaluation
Evaluation on a non-anonymous basis
• Unless otherwise specified in call
for proposal
• Register as an Evaluator
• https://cordis.europa.eu/emmfp7/
Role of Commission staff
• Commission staff may advise on:
Background on previously supported or on-going projects
Relevant supplementary information (directives,
regulations, policies, etc)
Evaluation rules
Key points within the Work Programme,
e.g. issues related to “Relevance”
• Commission staff may not introduce:
New elements (cannot fill in “gaps” in proposals)
Interpretations
Writing an FP7 proposal is NOT just a
creative process for A NICE IDEA
• It requires to SHOW scientific,
technological and depth
knowledge of the subject
• You must present references,
legislatures, previous work and
experience
Be precise and to the point
• Use drawings to show your
methodology at once
• Use bold types in phrases that you
would like to emphasize
• Present cohesion and interaction
among work packages
Overview of the Evaluation
Process
Submission
Full Proposal
Proposal
forms
Individual
reading
Evaluators
Criteria
Consensus
Panel
Evaluators
Evaluators
Criteria
Criteria
Finalisation
Final ranking
list
Rejection list
Proposals in
suggested
priority order
Eligibility
COMMISSION
COMMISSION
Role of experts
FP7 evaluation process
IAR 1
CONSENSUS MEETING
IAR 2
RECOMMENDANTION
IAR 3
CONSENSUS REPORT BY THE
RAPORTER
FOR FUNDING OR NOT
EVERY IAR IS 3 TO 4 HOURS
PANNEL MEETING – RESULTS ARE
SEND TO APPLICANT
0 - the proposal fails to address the issue under examination or can not be
judged against the criterion due to missing or incomplete information
1 - poor
2 - fair
3 - good
4 - very good
5 - excellent
RELEVANCE
QUALITY OF THE CONSORTIUM
1. CONCEPT – CAPABILITIES OF THE PARTNERS
QUALITY OF COORDINATION
QUALITY OF MANAGEMENT
MOBILISATION OF RESOURCES
2. ACTIVITIES – AND CAPACITY TO DO THE TASK
POTENTIAL IMPACT
3. THE TARGET GROUPS THAT BENEFIT
Process Evaluation - Individual reading
(Will be done remotely)
The experts:
• Evaluate the proposal individually
– without discussing with the other evaluators
• Check whether the proposal is ‘in scope’
• Complete an Individual Evaluation Report (IER) form giving
comments and scores on all criteria
• Sign and date the form
– IERs should be checked and, if necessary, returned with a
request to further justify the score given
– Scores must be in line with comments
Process Evaluation - Consensus (1)
• Built on the basis of the individual evaluations
• The aim is agreement on scores and comments
• Usually involves a discussion
– 1st part may be carried out remotely
• “Outlying” opinions need to be explored
– Not just a simple averaging exercise
– It is quite normal for individual views to change
• Moderated by a Commission staff-member
– helps the group reach a conclusion
– provides information if necessary
– does not contribute opinions
Process Evaluation - Consensus (2)
• A rapporteur is appointed, who is responsible for
drafting the consensus report (CR)
– includes consensus marks and comments
• The quality of the CR is paramount
– It is not often changed at the panel review stage
• The aim is:
– a clear assessment of the proposal, with justification
– clear feedback on weaknesses & strengths
• To be avoided:
– scores that don’t correspond with the comments
– recommendations in view of resubmission
The Panel Review
• EC ask some evaluators in each sub-activity to examine and
compare the CRs of every proposal that passes all thresholds
• Key function is to ensure consistency
• The Panel will recommend for a sub-activity a priority order
including final marks and comments for each proposal
Evaluation Summary Reports (ESR)
Any new scores (if necessary) … should be carefully justified
• Ranking of proposals with identical consensus scores
• Prioritise certain criteria?
• Consider overall balance?
• Budget?
• Clear guidance for contract negotiation
Process Commission Follow-up
•
•
•
•
•
•
•
•
Evaluation summary reports sent to applicants
– “initial information letter”
– Redress procedure
Draw up final ranking lists
Information to the Programme Committee
Contract negotiation
Formal consultation of Programme Committee (when required)
Commission decisions
Survey of evaluators
Independent Observers’ reports
New for
FP7
The evaluation criteria
• Criteria adapted to each funding scheme and
each thematic area
– specified in the work programme
• Three main criteria:
– S&T Quality (relevant to the topic of the call)
• Concept, objective, work-plan
– Implementation
• Individual participants and consortium as a whole
• Allocation of resources
– Impact
• Contribution to expected impacts listed in work
programme
• Plans for dissemination/exploitation
Process FP7 Evaluation Criteria
Applicable to ALL funding schemes
1. S/T quality
(in relation to the
topics addressed by the
call)
• Sound concept,
and quality of
objectives
2. Implementation
• Appropriateness of
the management
structure and
procedures
3. Impact
• Contribution, at
the European and /
or international
level, to the
expected impacts
• Quality and relevant
listed in the work
experience of the
programme under
individual
the relevant topic /
participants
activity
Process
Funding schemes

Collaborative projects
Support to research projects carried out by consortia with
participants from different countries, aiming at developing new
knowledge, new technology, products, demonstration activities
or common resources for research.
The size, scope and internal organisation of projects can vary
from field to field and from topic to topic.
Projects can range from small or medium-scale focused
research actions to large-scale integrating projects for
achieving a defined objective
Projects may also be targeted to special groups such as SMEs,
Specific International Co-operation Actions, etc.
Template of the IER for your information
(double click on the object)
29
Guidance for evaluators – Call FP7-ENV-2008-1
Process FP7 Evaluation Criteria
Collaborative projects
1. S/T quality
2. Implementation
3. Impact
(relevant to the topics
addressed by the call)
• Prgress beyond the
state-of-the-art
• Quality and
effectiveness of the
S/T methodology and
associated work plan
• Quality of the consortium
as a whole (incl.
complementarity, balance)
• Appropriate allocation and
justification of the resources
to be committed (budget,
staff, equipment)
•Appropriateness of
measures for the
dissemination and/or
exploitation of
projects results, and
management of
intellectual property.
Process
Funding schemes

Coordination & support actions
Support to activities aimed at coordinating or supporting
research activities and policies (networking, exchanges,
trans-national access to research infrastructures, studies,
conferences, etc).
same scope and objectives as in FP6
Template of the IERs for your information
(double click on the object)
31
Guidance for evaluators
– Call FP7-ENV-2008-1
Coordination & support actions
1. S/T quality
2. Implementation
3. Impact
(in relation to the topics
addressed by the call)
• Quality of the
ordination of high quality consortium as a whole
research
(including
complementarity and
CA • Quality and
balance)
effectiveness of the coordination mechanisms, • Appropriate allocation
and associated work plan and justification of the
• Contribution to the co-
SA
• Quality and
effectiveness of the
support action
mechanisms, and
associated work plan
resources to be
committed (budget,
staff, equipment)
• Appropriateness of
measures for
spreading excellence,
exploiting results, and
dissemination
knowledge, through
engagement with
stakeholders, and the
public at large.
RELEVANCE
Examines if the objectives of the work programme are met
The proposer must read in between lines
It must be clear what the commission wants from the call
Good practice: attend info day, inside information
Covering the objectives u get a 4, additional objectives might give u a 5
TIPS:
• Do not copy paste the objectives from the work programme
• U must copy and specialize them to the concept of the proposal
• FIND THE KEY WORDS (ie networking, knowledge transfer) relating the key
word to the objective
• TRY TO QUANTIFY OBJECTIVES ( i.E. Not just networking – but networking of
three clusters)
• Always make a direct reference under each objective with the related work
packages of how this objective will be achieved
Proposal relevance tips
• Identification of impact from Call
• Definition of strategic goals for our proposal
which clearly fulfill the expected.
• Definition of main actions/activities clearly
related to the strategic goals
• Work Packages for actions/activities.
QUALITY OF CONSORTIUM
Examines the experience of the consortium in relation to the objectives and the work
need to be done, complementarities between partners, suitability of work undertaken,
and geographical coverage
TIPS
Always include a description in the beginning of consortium description indicating the
role of each partner and their added value to the project.
It is preferred to have inside the consortium mix Member States, New member states
and Associated states indicating transfer of good practices to new member and
associated states.
Under each partner description indicating the excellences (previous projects, research
activities and experiences) and of the partner related to the proposal followed by the
CVs of key persons involved and their role in the project
These excellences should be summarized into an excellence key areas of the
consortium in relation to the work need to be done
Indicate the logic behind consortium geographical and context selection (why these
partners are selected)
The coordinator
• There is no limitation related to the size, nature, legal
status and the years of existence for the proposed
coordinator.
• In the management forms (A1, A2 and A3) the
turnover is presented. The is a indirect but very
important effect to the decision of the Commission.
QUALITY OF COORDINATION
Examines coherency of the proposal. If there is a methodology that defines the
interrelationships between the work packages and the work tasks
The plan must convince the evaluators that is a bottom down approach from the
implementation plan to the work packages
TIPS
• Start with the relation of the objectives to the work need.
• Define your methodology of work mentioning entities (not yet work packages)
• Define the interrelationships of the WPs and the work tasks inside the work packages
• Your methodology should have a clear description of impact creation in the target groups
and European Added Value through the actions of the dissemination plan
• Do not include management packages, just coordination packages this would be explained
in another section.
• YOU MUST ALWAYS START WITH THE METHODOLOGY OF WORK AND THEN
WRITE THE WORKPACKAGES
• Work packages should mirror the work in the implementation plan with CLEAR milestones
and derivables
• In general you could exceed the number of pages asked from the commission
General tips
• Every text with more than three paragraphs must
begin with a 2-3 lines abstract with a clear reference
to what the remaining text presents.
• In most cases the opinion of the evaluator is coming
from this abstract.
• Phrases 8 words, paragraphs max 3-4 lines.
QUALITY OF MANAGEMENT
Examines the experience and management mechanisms of the
management unit
The evaluator examines the capacity of the coordinator to handle the tasks
They like to have a strong centralized management and not very complex
structures
They like to see decision making mechanisms from steering committees
TIPS
Always start indicating the central management control indicating the experience
of the coordinator to undertake the tasks in similar projects
Illustrate decision making mechanism (simple but partner inclusive)
Show contingency plan with risks (i.e. what happens if partner withdraws)
Show IPR management
Show quality assurance mechanisms
Show knowledge management with in partners.
For each of the above relate them with management instruments (web sites,
consortium agreements, meetings, etc)
MOBILISATION OF RESOURCES
Examines the coherence of the financial plan to handle the tasks, relating to the
allocation of work in PM within work packages and among partners.
The financial plan must be delivered with the proposal although there are no predefined
forms from that (except A3, deliverable lists)
TIPS
ALWAYS state the CVs of key persons involved with the project and their role
Illustrate other resources for each organization that are allocated to the project in the
partnership description (ie equipment, previous work done etc)
Subcontracting is always examined in detail (why and who will do it?)
Avoid large differences in work allocation between partners
If a partner gets more than 30% of the budget (management and coordination) is
negative
Management no more than 7%
POTENTIAL IMPACT
Examines the impact (results) to target group defined, and the European
Added value and mainly your dissemination plan
Relates to the previous work done in European and National Level
TIPS
Always start with the clear definition of the proposal target groups indicating the
potential impact in quantitative terms if possible
Use structured bulleted writing and not abstract descriptions
The dissemination plan should provide evidence of European added value in
quantitative terms (not general instruments i.e. web page but exclusive actions such
as connection with specific initiatives, work groups etc)
Always illustrate impact in relation to previous work done – specific national and
European programmes indicating the approach mechanisms (even it is obvious)
The main evaluation criteria is the dissemination plan. The above issues should be
described as work to be done in the dissemination work package
ABSTRACT
Should start with objectives followed by the consortium logic
Then should describe the implementation plan with the expected results
and finish with the impact through the dissemination plan
Self-assessment process
• Most of failed proposals are taking a low mark in 1-2
evaluation criteria (relevance is the most usual).
• The pre-evaluation of the proposal by internal or
external executives (definitely not involved in the
proposal preparation process) based on the
published evaluation criteria.
Conclusion
• Become evaluator is the best way to learning
to write proposals
• Understanding how EC works
• Networking
• Well paid
Thank you!
Leonardo Piccinetti