Transcript NIH FUNDAMENTALS - Pennsylvania State University

The Grant Writing and
Review Process at NIH
Joshua Smyth
Professor of Biobehavioral Health and Medicine; Associate Director, SSRI
Rhonda BeLue
Associate Professor of Health Policy and Administration
Jenae Neiderhiser
Professor of Psychology
Danielle Downs
Associate Professor of Kinesiology and Obstetrics & Gynecology
Thanks to:

Lori Francis, Associate Professor of
Biobehavioral Health

Brittany Frost, Social Science Research
Institute
Workshop Outline




NIH Organization
NIH Funding Mechanisms
The Grant Writing Process
 Focus on the R01
The NIH Review Process




Overview of Review Meeting
The Scoring Process
A Penn State example
Workshop Evaluation
I. The NIH

Department of Health and Human Services

National Institutes of Health

25 Awarding Institutes/Centers aka ICs



e.g., NICHD, NIMH, NIDA, NIA, etc.
Center for Scientific Review
Office of the Director
The NIH Extramural Team
A. Program
B. Grants
Management
C. Review
A. The Program/Institute Staff
Program Administrator







Maintains knowledge of scientific area
Attends study section meetings
Makes funding recommendations
Monitors scientific progress
Identifies scientific area of importance
Reports to senior staff
Development of programs and initiatives
B. Grants Management




Interprets Federal regulations and policies
Assures compliance with Federal regulations
and policies
Monitors financial aspects of projects
Interprets regulations and policy
C. Review:
Scientific Review Group (1st Level)


Center for Scientific Review (CSR) or NIH
Institute & Center (IC)
Scientific Review Group (SRG)


Non-federal scientists with relevant expertise
Led by a Scientific Review Officer (SRO)

http://www.csr.nih.gov/Roster_proto/sectionI.asp
C. Review:
Advisory Council or Board (2nd Level)





The potential awarding IC performs the second level of
review
NIH program staff examine applications for impact
(formerly “priority”) scores, percentile rankings, &
summary statements against the IC’s needs
Program staff provide grant funding plan to Advisory
Council or Board
Advisory Council or Board advises the IC director
Director makes final decision
NIH Grant Application Cycle
Investigator
Institution
Initiates Research
NIH
Peer Review
Submits Application
Council
Review
Allocates Funds
Conducts Research
Funding
Decision
Grant Application:
It’s a process, not an event
1. Communicate with Program Officer

Introducing ideas, getting feedback, pre-review
2. Get your proposal to the right review committee



Review the rosters and talk to colleagues
Effectively wording the abstract
Make a written request
3. Seek feedback from colleagues and consultants on
drafts of the grant (prepare ahead!)
4. Consider who is likely to review your grant (review
the rosters) and make sure to know and cite their
work when relevant
5. Recognize that funding on first submission is rare
II. NIH Grant Mechanisms







Ks: NIH Career Development Awards (K01, K02, K05,
K07, K08, K22 [K99/R00])
P01: Research Program Project Grant
P30: Center Core Grants
R01: NIH Research Project Grant Program
R03: NIH Small Grant Program
R13: NIH Support for Conferences and Scientific
Meetings (R13, U13)
R15: NIH Academic Research Enhancement Award
(AREA)
NIH Grant Mechanisms
(continued)






R21: NIH Exploratory/Developmental Research Grant
Award
R34: NIH Clinical Trial Planning Grant
T series: NRSA Training Grants (T32, T34, T35, T90,
etc.)
U series: Research Project Cooperative Agreement
Diversity Supplements: Research Supplements to
Promote Diversity in Health-related Research
Roadmap: NIH Roadmap Initiatives (Director’s
Pioneer Award; Director’s New Innovative Program)
R01 (primary focus today)




Used to support a discrete, specified,
circumscribed research project
NIH’s most commonly used grant program
Body of the grant (Research Strategy) is 12
pages (mostly single spaced)
Generally awarded for 3 to 5 years

Up to $500,000/year without exceptions needed

Advance permission required for $500,000 or more
(direct costs) in any year
New and Early Stage Investigators:
A Competitive Edge


New Investigator has not previously served
as a PI for an R01; may have been an
investigator or received other smaller,
developmental or research training awards
Early Stage Investigator (ESI) is within 10
years of completing her/his terminal research
degree, or completing medical residency
III. The Grant Writing Process

Grant writing is:



A skill like any other…
But not the same skill as article writing
Instead, more of a problem-based writing
activity (theory and practice problem)
A few preliminary tips

Start early, make a timeline and STICK TO IT





Should allow time for serious pre-submission review
and subsequent revision
Develop a relationship with project officers
It is not possible to overdo clarity
Let your passion come through in your proposal
Take advantage of early stage and new
investigator opportunities
Getting ready to write


Know what has been done
Know what has been funded


NIH website RePORTER (formerly CRISP)
Decide on the problem

Important enough to get funded but simple enough
to explain as clean design


in 12 pp for the R01; less for some other mechanisms
Assemble team

Complementary skills; Seniority/competence/other
by association; People you can count on
Getting ready to write

Communicate with program officer



Establish a relationship and trust (funding
decision)
Acquire information on mechanism and
priorities
Obtain input on aims/proposal
Main Sections of the NIH Application
(see Francis et al. for some examples)




Face Page
Table of Contents
Performance Sites
Other information



Project Summary (Description)
Public Health Relevance Statement
Facilities & Resources
More Sections

Key Personnel


Budgets (for each study year)


Biosketches -- with personal statements
Budget Justification
Other sections (not discussed today);
for example:


Clinical Trial and Human Embryonic Stem
Cell (HESC)
List of Research Plan Attachments
Main Sections of the R01:
Specific Research Plan



[Introduction – revisions only]
Specific Aims: The basis for the proposal’s organization
Research Strategy


Significance and Innovation
Approach
 Preliminary studies
 Design
 Sample/recruitment/power analyses
 Procedures & measures
 Analyses
Page Limit Guide: Plan your
proposal with these limits in mind
Section of Application
Page
Limits
Introduction (for resubmission application only)
1
Specific Aims
1
Research Approach: R03, R13/U13, R21, R36, R41,
R43, Fellowships (F), SC2, SC3
6
Research Approach: R01, single project U01, R10, R15,
R18, U18, R33, R24, R34, U34, R42, R44, DP3, G08, G11,
G13, UH2, UH3, SC1
12
Biographical Sketch
4
Page limits may vary for other funding mechanisms.
Check Funding Opportunity Announcement:
http://grants.nih.gov/grants/guide/search_results.htm?scope=pa&year=active
And More Sections

Protection of Human Subjects







Women and Minorities
Planned Enrollment Table
Children
References Cited
Letters of Support
Resource Sharing Plan
Checklist
SO START EARLY!
General themes of success

Clarity is key – each point follows
naturally




Tell a (scientific) story
Everything that needs to be there, is;
nothing extra
Communicate your excitement
Get feedback early on

And often
General themes to R01 Success

Impact and Significance


Practice (2-3 sentences)
 Prevalence of problem in population
 Important social concern/public health problem
Theory (model)


Innovation


Building, testing, using
New directions, value added
Compelling Preliminary Research
Keys to Success – Specific Aims

Specific Aims may be only part read by some
reviewers


No more than one page
Tell the entire story



Functions as an abstract would in a manuscript
End with (actual) specific aims
Hypotheses and aims must align
Keys to Success – Background

Background and significance


This is where you ‘hook’ the reader on
your story


Used to be an explicit section; no longer
Functions as the ‘introduction’ does in a
manuscript
Likely no more than 2-3 pages
Preliminary Studies

Research team collaboration no longer
required to be documented


Still very helpful
Preliminary study ‘types’ – document:




Relevant previous research
Research that demonstrates competence in
requisite domains
Capacity to recruit in specific populations
and/or contexts
Pilot data specific to proposal
Keys to Success - Methods

Methods are very important





Overall -- clarity and detail
May include a table that traces aims to hypotheses
to constructs to measures (table/s)
Is the design feasible?
Are there gaps in the methods (e.g., fidelity for
interventions)
Statistics are essential – product must match aims


Consider a methodologist team member
Include a detailed timeline
And More Methods: The Sample

Preference for representative samples


Generalizability from a single entity
(university, clinic, state)



Students only if relevant to age/situation
(e.g., college drinking)
Justified exclusions
Unit of assignment is unit of analysis
Sample size/power analysis

For each outcome/planned test
Budgets

Direct Costs





Senior Personnel (PI, co-Is, project director) (PSU
fringe at 36.5%)
Other Personnel (full time staff, RAs, part time
wages) (PSU fringe 36.5% for full time staff, 13.2%
for Grad Assts AY; 7.9% for part time wages and
summer)
Equipment
Travel
Participant/Trainee Support Costs
Budgets, cont.

Other







Materials and supplies
Publication costs
Consultant services
Subawards/Consortium/Contractual Fees
Other
Indirect Costs (~50% at PSU, but does not
include all expenses)
Budget justification
Receiving the Summary
Statements: The Hardest Part!
1. Reviews critical, even harsh
2. Reviewers usually find grant’s weaknesses,
while recognizing strengths
3. Summary statements spend much more time
on critique than praise
4. Many investigators experience a mixture of
rage and depression when they read their
summary statements and easily lose
perspective
5. Take a day or two (or more!) and then read
again with a cooler head
Receiving the Summary
Statements: Bouncing Back!
1. Ask experienced colleagues to read reviews
2. Don’t interpret criticism as hopeless
3. Program Officer may be helpful in clarifying
critique
4. If “discussed” (rather than triaged), you
have a chance of funding in next round
5. The lower the initial score, the fewer
problems and more likely to be successful
after revision
Resubmission:
Resilience and Flexibility!
1. Persistence pays off in the grant
process!!
2. Second submission must respond to the
critiques through revision or clearly
defending reasoning
3. Same reviewers may or may not review
resubmission, but will see critique
Most Common Reasons for a
Poor Score (in priority order)








Lack of impact or significance
Lack of new or original ideas
Hypotheses ill-defined, superficial, lacking, unfocused, or
unsupported by preliminary data
Methods unsuitable, not feasible, not rigorous or not likely to yield
results; methods don’t clearly link to aims
Design not logical, inappropriate instrumentation, poor timing or
conditions; doesn’t link well to aims
Data management and analysis vague, not rigorous; analyses don’t
clearly link to aims
Inadequate expertise or knowledge of field for PI; too little time to
devote to the work
Poor resources or facilities; limited access to appropriate population
When to Revise




Basic idea was significant and
innovative or these can be bolstered
Design/measurement/analysis
problems can be clarified (more
information) or fixed
Need preliminary data
Problem is poor writing
IV. The NIH Review Process




A. The Review Meeting
B. Review Discussion
C. The Scoring Process
D. A Penn State Example
A. The Review Meeting:
The SRO’s Role Prior to Meeting





Point of contact until review group meets (then
project officer)
Analyze submissions for completeness and conflicts
Recruit ad hoc reviewers as needed
Schedule 1-2 day meeting
Assign applications to reviewers (at least 3)


Primary, secondary, discussant
Create review order based on preliminary impact
scores from best to worst within categories
Reviewers’ Role Prior to Meeting


Familiarize self with criteria, mechanisms, and scoring
Review assigned applications







Assign scores to each criteria and other areas
Write bulleted strengths and weaknesses for each criteria
Reviews are advice to institutes for funding decisions, not
advice to PI
Post scores and comments on NIH Commons
Read other reviews of assigned applications
Prepare presentation of reviews
Skim/read non-assigned applications
Format of the Review Meeting





SRO opening remarks
Chair orientation
New investigator R01 grants
Other R01 grants
Other grant types (R03, R15, R21, R34)

Applications discussed in order of Impact
Score; bottom 50% are not discussed
SRO Opening Remarks



Confidentiality
Review order
Proposals below median within each
category may not be discussed
Chair Orientation

Start with reviewer impact scores





May differ from posted scores
Goal of discussion is to clarify not reach
agreement
If scores are similar, shorter discussion
If scores are dissimilar, longer discussion
Recommended time



Primary – 5 minutes
Secondary – 3 minutes
Discussant – 2 minutes
B. Review Discussion




Identify proposal
Members in conflict leave
Reviewers provide preliminary impact scores
Reviews





Impact, Significance, Investigators, Innovation, Approach,
Environment
Stress main points, do not repeat previous points
Non-reviewers typically ask questions to clarify
Human Subjects issues affecting scoring
Open discussion to entire committee
Review Discussion (continued)





Ask for reviewers impact scores again
Identify the reviewers’ recommended range
Ask if anyone wants to score outside the
range
Entire committee records impact score
Discuss budget and other issues
C. The Scoring Process
1. Overall Impact Score: likelihood project will “exert a
sustained, powerful influence on the research field(s)
involved (1-9 scale)
2. A separate 1-9 score for each of 5 core criteria
(Significance, Investigators, Innovation, Approach,
Environment)
3. Additional review criteria help determine scientific and
technical merit BUT are not scored separately
4. Additional review considerations are addressed by
reviewers, but are not scored & are discussed after
group scores
Score Criteria

Overall Impact: will project exert a sustained, powerful
influence on the research field(s) as indexed by 5 core
review criteria
1. Significance: important problem addressed; how will this
improve scientific knowledge, technical capability, and/or
clinical practice
What is the Difference Between
Impact and Significance ?

Impact Addresses:


Probability of whether the research will exert a
sustained, powerful influence on the research field
Significance Addresses:


Does the project address an important problem or a
critical barrier to progress in the field?
If the aims are achieved, how will scientific
knowledge, technical capability, and/or clinical
practice be improved?
Score Criteria (continued)
2. Investigators: PI & other researchers well suited to the
project; appropriate experience & training; ongoing record
of accomplishments; complementary & integrated
experience; leadership approach, governance, and
organizational structure appropriate for project
3. Innovation: the work challenges and seeks to shift
current research or practice paradigms; utilizing novel
theory, approaches or methods, instrumentation, or
interventions; the work is novel
(Be innovative, but maybe not too innovative…)
Score Criteria (continued)
4. Approach: strategy, methodology, analyses are wellreasoned and appropriate; potential problems & alternative
strategies thought through; benchmarks set; risk is
managed
Most common reviewer complaint is lack of detail here
Typically the longest section
5. Environment: the environment will contribute to the
project’s success; institutional support, equipment, & other
resources sufficient; unique features of the environment,
subject population, collaborative arrangements
Additional Review Criteria (not scored)
Human Subjects:
1.
2.
3.
4.
5.
Protection of human subjects
Data safety monitoring plan (clinical trials only)
Inclusion of women, minorities, children
Vertebrate animals
Biohazards
Additional Review Considerations
1.
2.
3.
4.
Budget and period of support
Select agent research (infectious agents)
Applications from foreign organizations
Resource sharing plans
CRITERIA SCORING SYSTEM
HIGH
1.
Exceptional: Exceptionally strong with essentially no weaknesses
2.
Outstanding: Extremely strong with negligible weaknesses
3.
Excellent: Very strong with only some minor weaknesses
MEDIUM
4.
Very Good: Strong but with numerous minor weaknesses
5.
Good: Strong but with at least one moderate weakness
6.
Satisfactory: Some strengths but also some moderate
weaknesses
Low
7.
Fair: Some strengths but with at least one major weakness
8.
Marginal: A few strengths and a few major weaknesses
9.
Poor: Very few strengths and numerous major weaknesses
CRITERIA SCORING SYSTEM
(continued)



Minor Weakness: An easily addressable
weakness that does not substantially
lessen impact
Moderate Weakness: A weakness that
lessens impact
Major Weakness: A weakness that
severely limits impact
CRITERIA SCORING SYSTEM
(continued)
Final Overall Impact Score:
Mean of all reviewers’ final impact scores X 10
Range = 10 (high impact) -- 90 (low impact)
NOTE: Scoring likely to produce applications with
identical scores (“ties”).
Thus, other factors (e.g., mission relevance, portfolio
balance) will be considered when all other things are
essentially equal
Key Sections in R01 Proposal Format

Research Plan Components



Specific Aims
Research Strategy
 Includes Background & Significance; Preliminary
Studies/Progress Report; Research Design & Methods
Facilities and Equipment

Reflects the Environment criterion


For ESIs should describe the institutional investment in the success of the
investigator
Biographical Sketch [NEW! as of Jan 25th, 2015]


Personal statement – why well-suited for project, 4 pubs
Contribution to science - you describe up to 5 of your most
significant scientific contributions (<=1/2 page each); up to 4
pubs or other products for each contribution area
D. A Penn State Example
Mock NIH Study Section
Chair: Joshua Smyth, Associate Director, SSRI
Reviewer #1: Rhonda BeLue, Associate Professor
of Health Policy and Administration
Reviewer #2: Jenae Neiderhiser, Professor of
Psychology
Reviewer #3: Danielle Downs, Associate Professor
of Kinesiology and Obstetrics & Gynecology
Review group members: Workshop attendees
Links of Interest
Enhancing Peer Review Criteria:
http://grants.nih.gov/grants/guide/notice-files/not-od-09025.html
Page Limits:
http://grants.nih.gov/grants/forms_page_limits.htm
Human Subjects:
http://grants.nih.gov/grants/policy/hs/index.htm
SF424 guidelines for submission:
http://grants.nih.gov/grants/funding/424/index.htm
Glossary: http://grants.nih.gov/grants/glossary.htm
General Resources
NIH RePorter (formerly CRISP)
https://libraries.ucsd.edu/info/resources/nih-reporterformerly-crisp
Revised Applications
http://grants.nih.gov/grants/policy/amendedapps.htm
NIH Grant Writing Tip Sheets
http://grants.nih.gov/grants/grant_tips.htm
Getting an RO1
http://sciencecareers.sciencemag.org/career_development/
previous_issues/articles/1190/getting_an_nih_r01
More general resources
NSF Proposal Writing
http://www.cs.cmu.edu/~sfinger/advice/advice.html
Other Proposal Writing Guides
http://www.learnerassociates.net/proposal/
Reasons Proposals Fail
http://chronicle.com/article/How-to-Fail-in-GrantWriting/125620/--let
New and Early Stage Investigator
Resources
New and Early Stage Investigators
http://grants.nih.gov/grants/new_investigators/
NIH Websites
http://public.csr.nih.gov/Pages/default.aspx
http://grants.nih.gov/grants/oer.htm (forms,
grant search, etc.)
SSRI Listserv


New subscribers can join the SSRI
listserv by sending mail to:
[email protected]