Transcript Stage 2

Foundation Scheme Pilot 1:
What? Who? When? How?
Lori Burrows, PhD
CIHR University Delegate
June 12, 2014
What?
First Pilot competition of CIHR’s new Foundation Scheme:
5-7 year grants aimed at scientific LEADERS.
If successful, you cannot hold other CIHR open grants
as PI during the span of your Foundation grant. You can
be a co-investigator on other Foundation or Project grants.
Existing grants will be rolled into your Foundation grant. If
unsuccessful, you WILL NOT LOSE EXISTING GRANTS.
Many elements of this competition remain uncertain,
because they are still in development and may yet change
based on the results of ongoing pilots.
example of ‘still in development’
“The adjudication criteria considered at Stage 1 and 2 are different. A
detailed description of these criteria can be found within the CIHR
Adjudication and Selection Process Manual for the 2014 Foundation
Scheme Competition. (link to come)”
Money available for 2014/15
The combined total amount available for 2014-15 Transitional
OOGP and Foundation Pilot is ~$500M. The two competitions
overlap for the first time…unclear how this will play out in
terms of total number funded.
120 to 250 Foundation grants may be funded; ‘most
Foundation grant budget requests will fall within a range of
$50K to $1.5M per annum’.
The budget request should be consistent with the applicant's
previous CIHR open grant research funding history.
TIP: You can point out that your historic request was higher than the
amount you actually got, due to ATB cuts.
Who?
You must fit CIHR’s ‘independent researcher’ definition &
a) be the PI or co-PI on an open CIHR grant that ends
between Oct 1, 2014 and Sept 30, 2015, or
b) be an established investigator (>5 yr since 1st appt) who
has never held a CIHR open grant as PI or co-PI, or
c) be a new investigator (<5 yr since 1st appointment);
leaves of absence aren’t included, and part time
appointments will be pro-rated up to the maximum of 60
months post-appointment
CIHR will verify your eligibility at registration.
What are defined as ‘open’ grants?
Open Operating Grant Program (OOGP)
Partnerships for Health System Improvement (PHSI)
Knowledge Synthesis Grant (KRS)
Knowledge to Action Grant (KAL)
Proof of Principle Program Phase I and II (POP I and POP II)
Industry-Partnered Collaborative Research (IPCR)
New names for PI and co-I
Program Leader: individual responsible for setting and
overseeing the intellectual and strategic direction of the
research. Non-Canadian PLs must have Canadian co-Leader.
Program Leader – Administrative Coordinator: must be a
PL, at Canadian institution; assumes all responsibility for
administrative matters related to the grant including submitting
the application and coordinating any required reporting
Program Expert: participates in or contributes to the program
of research but does not direct the program; may or may not
be involved for the entire length of the grant; may or may not
be named on the grant at discretion of PL
TIP: CVs of Experts are not included or assessed in Foundation
applications, and they are not named in funding decisions. They can
document their involvement in Foundation grants in their CVs, reports,
etc., as agreed upon with the Program Leader(s), and they can apply for
Project grants.
When?
Registration – June 23, 2014
Stage 1 Application – Sept 15, 2014
Stage 1 Decision – Dec 1, 2014
Stage 2 Application – Feb 5, 2015
Stage 2 Decision – May 15, 2015
Stage 3 Decision – July 2, 2015
Funding Start – July 1, 2015
(yes, before the stage 3 decision…)
How?
1. Register via ResearchNet by June 23, 8 pm
2. If you pass the eligibility test, CIHR will send a link for
the stage 1 application, due Sept 15, 2014, 8 pm
3. If unsuccessful at stage 1 (decision in Dec 2014),
you can apply to the Spring 2015 Transitional OGP,
registration deadline Jan 2015.
4. If successful at stage 1, you can apply to stage 2 OR
to the Spring TOGP…not both.
TIP: Depending on the comments at stage 1, and projected success
rates at stages 2 and 3, you may want to consider whether you go
forward, or bail out to the TOGP. If you go forward to stage 2 and are
unsuccessful, you will have to wait until Fall 2015 to re-apply, with no
money until July 2016 even if you succeed the second time.
2014
2015
NO (87%?)
NO (87%?)
Mar/14
OOGP
chose
to
renew
early
Sept/14
Fdn
Pilot
$ Oct/14
choose to
skip Fdn
Mar/15
transitional
OOGP
NO
YES (13%?)
$ July/15
decided
not to go
forward
to stage 2
Feb/15 YES
$ July/15
stage 2
(May/15)
MAYBE
May/15
stage 3
(July/15)
choose
to
renew
early
Sept/15
Fdn
Pilot
NO –
REAPPLY
NO –
REAPPLY
choose to
Mar/16
Project
skip Fdn
NO
YES (13%?)
Sept/14
stage 1
(Dec/14)
YES
2016
YES
Sept/15
stage 1
(Dec/15)
$ July/16
decided
not to go
forward
to stage 2
YES
Feb/16 YES
stage 2
(May/16)
MAYBE
May/16
stage 3
(July/16)
YES
YES
$ July/15
$ July/16
$ July/16
NO –
REAPPLY
NO –
REAPPLY
Sept/16
Fdn
Sept/16
Project
YES
$ April/17
How do I register?
http://www.cihr-irsc.gc.ca/e/48053.html
You need a ResearchNet account, a CIHR
PIN and a Common CV account.
the Common CV
• fill out the CIHR version of the Common CV
TIP: this can take significant time!
• to register for the pilot, pick CIHR as the agency and
‘registration’ for the format
• validate it and and click ‘submit’ (it doesn’t go
anywhere)
• the system will generate a confirmation number that
needs to be entered into the registration application
on ResearchNet
• if you change anything, you need to repeat this
process, the confirmation number will change
on ResearchNet…
• Under ‘Apply for Funding’, pick ‘Foundation Scheme….’
and follow the instructions.
• The Program Leader(s) must not change between
registration and application.
• You must provide a 5 (new investigators) or 7 yr
budget estimate. You don’t need to justify at this point,
and you can change it at stage 2.
TIP: For new people, look at CIHR’s funding database for historic
amounts handed out for the type of work you’re proposing to do
• Suggest reviewers. This is critical, as these people
will be recruited into the College of Reviewers. Exclude
reviewers you don’t want.
What’s my competition like?
• as of June 6, 43 registrations completed, 874 in
progress
• 305 are current grant holders; 278 have never held
CIHR grants as PI (might be some overlap); 267 are
new investigators and 67 are undefined
• includes registrations opened on spec…but CIHR
expects more at the last minute
• CIHR is expecting ~1200 bona fide applicants,
despite high numbers of registrants (since many will
register but not submit). But, they are surprised at the
new investigator numbers so far…
Stage 1 application (in development)
Detailed instructions (soon) available on
ResearchNet under ‘Opportunities’, ‘Foundation
Scheme’.
“To complete your Stage 1 Application, follow the instructions
in the Foundation Scheme: 1st Live Pilot ResearchNet
"Application" Phase Instructions. (link to come)”
Stage 1 application content
• Program Leader(s) will need to generate a
Foundation Scheme CV using the Common CV
• SHORT stage 1 application has 3 sections:
– Summary (1 page)
– Caliber of the Applicant (1.5 pages, 75%)
• -Leadership (1/2 page, 25%)
• Significance of contributions (1/2 page, 25%)
• Productivity (1/2 page, 25%)
– Vision and Program Direction (1 page, 25%)
Program Leader’s CV
http://www.cihr-irsc.gc.ca/e/48206.html
See website for details of what is to be included. Only
select contributions, with maximum numbers per section.
e.g. Publications (Most relevant – up to 25)
‘Program Leaders will provide information regarding their
publications in the form of journal articles, books, book
chapters, reports, manuals, clinical care guidelines, and/or
conference publications’.
Summary (1 page)
• summarize research experience and vision of the
research program:
– overarching focus of research career, major questions or
issues explored, and the impact of the research;
– expertise and experience;
– Important collaborations, within or outside of the research
community; and
– vision or direction of the program of research to advance
knowledge and/or its application to health care, health
systems, and/or health outcomes.
TIP: the Summary is a MARKETING DOCUMENT – SELL YOURSELF!
Leadership (1/2 page)
• highlight significant and effective leadership
experience, reputation in your field and evidence
that you’ve led major projects or research
programs – point to examples in your CV
• reviewers are asked if you are widely
recognized, influential, community-building; if
you have established, resourced, directed major
efforts including training, infrastructure
development, collaborations
Significance of Contributions (1/2 page)
• highlight your contributions and their impact,
tangible benefits or positive influences – give
examples from your CV
• reviewers are asked if you have significantly
advanced knowledge and/or its translation into
health care/systems/outcomes; and if you’ve
engaged/trained/launched the careers of promising
individuals in research or health-related areas
TIP: although the sections may have overlap in terms of what you point
to, CIHR thinks that true Leaders will have so many examples of their
awesomeness, they won’t need to repeat any of them
Productivity (1/2 page)
• highlight your productivity – quality and
quantity of outputs, with examples from CV
• reviewers are asked if you have
demonstrated an outstanding level of
research outputs based on prior work; if your
previous work generated high quality
research outputs
TIP: if there is more than one Program Leader, their joint
productivity will be considered
Vision and Program Direction (1 page)
• articulate a compelling vision and direction for the
program…highlighting the goal, overall objective(s),
expected outputs/contribution(s), and significance of
the proposed program. Specific details for each
project in the program are not expected, because
the grants are meant to be flexible, to allow
innovation and new directions.
• reviewers are asked if program is well-defined and
well-articulated, logical career progression for the
Program Leader(s)? forward-looking, creative, and
appropriately ambitious? Does the vision aim to
significantly advance knowledge and/or its
translation?
Stage 2 application
• get your stage 1 reviews and an invitation to go
forward (or not)
• the stage 2 application is 12 pages total plus ½ page
of budget justification
• will include:
– summary (1 page)
– quality of program – 40%
• research concept (3 pages, 20%)…coherence, significance
• research approach (2 pages, 20%)…appropriate, flexible
– quality of expertise, experience, resources – 60%
• expertise (3 pages, 20%)…who are your Experts?
• mentorship and training (2 pages, 20%)…comprehensive plan
• quality of support environment (1 page, 20%)…appropriate
– budget justification (1/2 page)
Peer review of your stage 1 application
This critical component remains under development.
You will suggest (and exclude) potential reviewers upon registration.
CIHR is shooting for 5 reviewers per application; some may be selected
from existing peer review committees. Eventually, reviewers will be
members of the College of Reviewers.
Each reviewer will have up to 20 applications, and after scoring the
criteria (and resolving discrepant scores via online discussion with
dissenting reviewers), will rank the applications (no ties allowed). Each
application will have a combined rank that is an average of 5 individual
rankings, and the combined ranks are used to make decisions at stage 1.
Right now, estimates are that the top ~350 may go forward to stage 2.
Pilot of the Structured Review Forms: Adjudication Scale
Stage 1 Reviewers
Stage 2 Reviewers
Stage 1 Reviewers stated that
they used the full range of the
adjudication scale; however,
Stage 2 Reviewers disagreed
Although the majority of Stage
1 Reviewers stated that the
adjudication scale allowed the
description of meaningful
differences, their comments
indicated that this was not true
Limitation of Stage 2 Reviewer
comments is that they only saw
the top 39 (of 77) applications
Distribution of Ratings: Letter Scale vs. Current Number System
% A’s Given out % B’s Given out
by Reviewers
by Reviewers
% C’s Given out % D’s Given out % E’s Given out
by Reviewers
by Reviewers
by Reviewers
Quality of Idea
41.15%
35.16%
16.93%
4.69%
2.08%
Importance of Idea
35.68%
39.84%
17.19%
5.99%
1.30%
Approach
24.74%
34.90%
23.44%
13.80%
3.13%
EER
42.45%
39.84%
12.50%
4.43%
0.78%
Average
36.01%
37.44%
17.52%
7.23%
1.82%
4.5 – 4.9
(A)
4.0 – 4.49
(A)
3.5 – 3.99
(B)
3.0 – 3.49
(C)
≤ 2.99
(D/E)
Scientific Merit
16%
20%
35%
24%
6%
Potential Impact
29%
29%
25%
12%
4%
Final Score
12%
13%
30%
34%
10%
25%
new:
A+B
73.5%
old:
A+B
55%
*Scientific Merit and Potential Impact based on Spring 2013 Knowledge Synthesis competition consensus score
Different scale (ABCDE is too close to letter grades in school)?
Could also be a problem for international reviewers as the letters mean different things in different countries.
Add granularity to the scale (e.g., A+, A, A-)?
5 letters is interpreted as a 20% difference between letter grades and this is not sensitive enough to capture
subtle differences between applications;
Stage 1 - Preliminary Reviews
• A little more than half of
the Stage 1 Reviewers
read other reviewer’s
reviews
• Reading other reviewer’s
reviews did not often
influence a reviewer’s
assessment of the
application
• In future surveys, this question will be re-framed to:
– Ask reviewers whether the ability to read other reviewers’ reviews was
useful/helpful
– Differentiate the responses between reviewers who had divergent
ratings/comments and those who had similar ratings/comments
28
Stage 1 Asynchronous Online Discussion
75% of reviewers did not
participate in online
discussions…
…and not because there was
nothing to say. Almost half
were not finished with their
reviews in time.
Reviewers felt their opinion was not
very influential, but about half
thought others could at least
occasionally change their
assessment. (Not clear if these were
the 25% that did participate).
Stage 2: Face-to-Face Meeting
Most thought the
face-to-face meeting
was necessary
All of the applications were listed in
rank order on a monitor (the top half
from Stage 1) and a funding cutoff
line was indicated, so people had an
idea of which apps needed
discussion
Applications were grouped into A
(green zone), B (grey zone) and C
(red zone), and the discussion
centered on group B apps.
Overall Satisfaction with Structured Review Process
Applicant satisfaction with the
structured review process
correlated with the state of the
application (funded, moved to
Stage 2, unsuccessful after
Stage 1)
Most Stage 1 Reviewers were
satisfied with review process
Most Stage 2 Reviewers were
satisfied with review process
31
Registration – June 23, 2014
Stage 1 Application – Sept 15, 2014
Stage 1 Decision – Dec 1, 2014
TOGP registration – Jan 2015
TOGP application – Mar 2015
Stage 2 Application – Feb 5, 2015
Stage 2 Decision – May 15, 2015
Stage 3 Decision – July 2, 2015
If you have questions, comments or concerns, email me!
[email protected]