Transcript A “Peer” into the NIH Review Process

The NIH Peer Review
Process
Sally A. Amero, Ph.D.
NIH Review Policy Officer
Office of Extramural Research
2010 NIH Regional Seminars
The NIH Peer Review Process
National Institutes of Health
• Primary Federal agency in the US
for conducting and supporting biomedical research
• 27 Institutes & Centers (ICs)
 Extramural and intramural programs
 24 ICs have funding authority
2
2
The NIH Peer Review Process
• Two-tiered process:
– Initial peer review
• Scientific Review Groups (SRGs)
• “Study Sections”
– I/C Advisory Council or Board (Council)
• Per year:
– Nearly 80,000 applications
– Over 18,000 reviewers
3
3
The NIH Peer Review Process
Application received
Assignments made


Initial peer review
Funding considerations
SRG; study section
Institutes or Centers (ICs)
IC or CSR*
Duals possible
Scientific Review Officer
Program Officer


Second level of review

Funding decisions
Council or Board (IC)
IC Director

Award!
Overview
*CSR = NIH Center for Scientific Review
4
4
The NIH Peer Review Process
Division of Receipt and Referral, CSR
• Check for completeness
• Determine area of research
• Assign an identification number
• Assign a grant number
• Assign to NIH IC for possible funding
• Assign to a Scientific Review Group
CSR = Central receiving point for all competing
applications submitted to the NIH
5
5
The NIH Peer Review Process
Referral to an SRG
CSR Review
Institute/Center Review
• Most R01s, fellowships, and
small business applications
• Some Program
Announcements (PAs,
PARs)
• Some Requests for
Applications (RFAs)
•
•
•
•
IC-specific features
Program projects
Training grants
Career development
awards
• RFAs
Review locus is stated in Funding Opportunity Announcement
6
6
The NIH Peer Review Process
To Request a Scientific Review Group
Cover letter of application
– Application title
– FOA # and title
– Request:
– Assignment to particular SRG or study section
– Assignment to particular IC for funding consideration
– Disciplines involved, if multidisciplinary
– Explanation for late application
Not all requests can be honored.
7
7
The NIH Peer Review Process
Cover Letter of Application
Instructions
– List one request per line
– Place SRG & IC review requests on separate lines
– Place positive & negative requests on separate lines
– Include name of IC or SRG, followed by a dash and acronym
– Explain each request in a separate paragraph
8
8
The NIH Peer Review Process
Confirmation of Receipt
• Email notifications
– eSubmissions: Grants.gov and NIH
– Paper submissions: Email to confirm assignment
• eRA Commons (http://era.nih.gov/commons/index.cfm)
– Grant application number
– SRG assignment
– SRO contact information
– Funding IC(s) assignment(s)
– New Investigator or Early Stage Investigator status
– Council round
– Multi-PD/PI status
9
9
The NIH Peer Review Process
Information
• Center for Scientific Review:
http://cms.csr.nih.gov/PeerReviewMeetings/CSRIRGDescription/
http://www.csr.nih.gov/committees/rosterindex.asp
• Institutes and Centers:
http://era.nih.gov/roster/index.cfm
• Areas of IC interest:
http://www.nih.gov/icd/index.html
10
10
The NIH Peer Review Process
Scientific Review Officer (SRO)
First level of peer review
– Designated Federal Official
– Extramural scientist administrator
– Identifies and recruits reviewers
– Manages conflicts of interest
– Oversees arrangements for review meetings
– Presides at review committee meetings
– Prepares and releases summary statements
11
11
The NIH Peer Review Process
Peer Reviewers
Recruitment
SRG rosters are posted 30 days
– Expertise
before the SRG meeting!
– Stature in field
http://era.nih.gov/roster/index.cfm
– Mature judgment
– Impartiality
– Managed conflicts of interest
– Balanced representation
• Gender
• Geography
• Diversity
• Seniority
– Availability
12
12
The NIH Peer Review Process
Types of Scientific Review Groups (SRGs)
• “Chartered” SRGs
– Multiyear terms
– Formal appointment process
– May include temporary members for special expertise
• Special Emphasis Panels (SEP)
– Ad hoc membership
– Often meet only once
13
13
The NIH Peer Review Process
Types of Reviewers
• Regular reviewers
– Written critiques, criterion scores, preliminary
impact/priority scores
– Final impact/priority scores
• Other Contributing Reviewers
– Written critiques, criterion scores, preliminary
impact/priority scores
– Cannot submit final impact/priority scores
14
14
The NIH Peer Review Process
Reviewer Assignments
• ≥ Three qualified reviewers (2 + 1)
• Based on scientific content of application
• Expertise of reviewer
• Suggestions from PI on types of expertise –
not names!
• Suggestions from Program staff
• Suggestions from SRG members
• Managing conflicts of interest
• Balancing workload
15
15
The NIH Peer Review Process
Conflicts of Interest (COI)
COI between a reviewer and an application:
– Financial
– Employment
– Personal
– Professional
– SRG membership
– Other interests
Two COI vouchers are submitted by each SRG member.
• Pre-meeting and Post-meeting
16
16
The NIH Peer Review Process
Scientific Review Groups (SRGs)
Make recommendations, not funding decisions!
– Scientific and technical merit
– Budget and project duration
– Bars to award – human subjects,
vertebrate animals, biohazards
– Resource Sharing Plans
– Other administrative factors
 Impact/priority scores
 Criterion scores
 Written critiques
17
17
The NIH Peer Review Process
Scientific Review Groups (SRGs)
Confidentiality
– All materials, discussions, documents retrieved or
– destroyed (except those in the public domain)
– Reviewers sent guidance with applications
– Application information secure
– All questions must be referred to SRO
– Meetings closed to the public
– Program staff may observe
18
18
The NIH Peer Review Process
Overall Impact/Priority Score
Reflects the reviewers’ assessment of the likelihood
for the project to exert a sustained, powerful influence
on the research field(s) involved
• Five (or more) scored criteria
• Additional review criteria
• Additional review considerations
19
19
The NIH Peer Review Process
Enhanced Review Criteria – Research Grants
Overall Impact
– Scored criteria:
• Significance*
• Investigator(s)*
• Innovation*
*Receive individual criterion scores
• Approach*
See Side-by-side comparison
• Environment*
(http://grants.nih.gov/grants/peer_review_process.htm)
– Additional review criteria
– Additional review considerations
20
20
The NIH Peer Review Process
Impact and Significance
Overall Impact
Reviewers will provide an overall impact score to reflect their
assessment of the likelihood for the project to exert a sustained,
powerful influence on the research field(s) involved, in
consideration of the (following) five core review criteria, and
additional review criteria (as applicable for the project proposed).
21
21
The NIH Peer Review Process
Impact and Significance
Significance
– Does the project address an important problem or critical
barrier to progress in the field?
– If the aims of the project are achieved, how will scientific
knowledge, technical capability, and/or clinical practice be
improved?
– How will successful completion of the aims change the
concepts, methods, technologies, treatments, services, or
preventative interventions that drive this field?
22
22
The NIH Peer Review Process
Impact and Significance
Overall Impact
– Not a sixth review criterion
– Not necessarily the arithmetic mean of the criterion scores
– Is the synthesis/integration of the scored and additional criteria
– Reviewer(s) assess the likelihood for the project to exert a
sustained, powerful influence on the research field(s) involved
– Likelihood (i.e., probability) is primarily derived from the
investigator(s), approach and environment criteria.
– Sustained powerful influence is primarily derived from the
significance and innovation criteria.
23
23
The NIH Peer Review Process
Impact and Significance
Significance
– Is evaluated and scored independently of other criteria
– Evaluation assumes that the “aims of the project are achieved”
and/or will be “successfully completed.”
– Reviewers should evaluate within the context of a
(research) field(s).
– Research field(s) may vary widely, so it would be helpful if
reviewers identify in their reviews the research field(s) within
which the project addresses an important problem or
critical barrier to progress.
24
24
The NIH Peer Review Process
“Review Criteria at a Glance”
(http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)
• Additional Review Criteria
– Are factored into overall impact/priority score
– Not assigned individual scores
• Additional, specific criteria for mechanism, or FOA
• Protections for Human Subjects from Research Risks
• Inclusion of Women, Minorities & Children
• Vertebrate Animals
• Biohazards
• Resubmissions
• Renewals
• Revisions
25
25
The NIH Peer Review Process
“Review Criteria at a Glance”
(http://grants.nih.gov/grants/peer/reviewer_guidelines.htm)
Additional Review Considerations
– Not factored into overall impact/priority score
– Not assigned individual scores
• Additional, specific considerations
• Applications from Foreign Organizations
• Select Agents Research
• Resource Sharing Plans
• Budget & Period of Support
26
26
The NIH Peer Review Process
Enhanced Review Criteria – Fellowships
Overall Impact/Merit
– Scored criteria
• Fellowship Applicant
• Sponsors, Collaborators, and Consultants
• Research Training Plan
• Training Potential
• Institutional Environment & Commitment to Training
27
27
The NIH Peer Review Process
Enhanced Review Criteria – Fellowships
Overall Impact/Merit
–Additional Review Criteria
• Protections for Human Subjects from Research Risks
• Inclusion of Women, Minorities, & Children
• Vertebrate Animals
• Biohazards
• Resubmissions
• Renewals
– Additional Review Considerations
• Responsible Conduct of Research
• Select Agents Research
• Resource Sharing Plans
• Budget & Period of Support
28
28
The NIH Peer Review Process
NIH Scoring System
• New system
– Numerical scores
•1.0 (exceptional) to 9.0 (poor)
• Final impact/priority score =
average of individual scores x 10
• Individual criterion scores
• Ranked by percentile for certain mechanisms
– Not Discussed (ND) - streamlining
– Other designations (NR, DF, AB, NP, etc.)
Final impact/priority scores range
from 10 through 90!
29
29
The NIH Peer Review Process
NIH Scoring System
• Preliminary scores (before the SRG meeting)
– Entered by assigned reviewers and discussants in
secure website
– Made available to other SRG members
• Final overall impact/priority scores (at the SRG meeting)
– Voted by private ballot
– All eligible SRG members vote
Reviewers are instructed to revise their
criterion scores after the meeting.
30
30
The NIH Peer Review Process
New Score Descriptors
Impact
High Impact
Moderate
Impact
Low Impact
31
Score Descriptor
1
Exceptional
2
Outstanding
3
Excellent
4
Very Good
5
Good
6
Satisfactory
7
Fair
8
Marginal
9
Poor
31
The NIH Peer Review Process
Streamlining
• Allows discussion of more meritorious applications
– Research projects ~ 50%
– Shared instrumentation ~ 40%
– Fellowship applications ~ 30%
– RFAs – pre - arranged limits
• Requires full concurrence of SRG
• Not discussed at SRG meeting
• Summary statement:
– Reviewer critiques
– Individual criterion scores
– No final overall impact/priority score (ND)
32
32
The NIH Peer Review Process
Streamlining
• Score order of review
– SRG discusses most meritorious applications first
– Entire SRG decides when to stop, which applications will
not be discussed in panel
• Other order of review (e.g., IC assignment,
mechanism)
– SRO prepares a list of average preliminary scores
– Distributes to SRG
– Entire SRG decides which applications to discuss
33
33
The NIH Peer Review Process
Pre-Meeting SRG Procedures
SRO
– Performs administrative review of applications
– Recruits reviewers, arranges for meeting date and site
– Assigns 3 SRG members to each application
– Makes applications available to reviewers
• Internet Assisted Review (IAR) site or on CDs
• Usually about six weeks before the SRG meeting
– Instructs reviewers in review procedures
– Monitors posting of initial scores and critiques in IAR
34
34
The NIH Peer Review Process
Reviewer Instructions and Guidelines
• Reviewer Orientation
• Review Criteria at a Glance
• Overall Impact versus Significance
• Scoring System and Procedure
• Critique Template Instructions
• NIH Certification Forms
• Budget Information
• Human Subjects, Protection and Inclusion
• Vertebrate Animals
• Resource Sharing Plans
Available at:
http://grants.nih.gov/grants/peer/reviewer_guidelines.htm#general_guidelines
35
35
The NIH Peer Review Process
Structured Critiques
New summary statement format
–
–
–
–
–
–
–
–
36
Bulleted comments from reviewers
Less text
Criterion scores
Decreases variability
Increases quality of information in critiques
More succinct, better organized
Encourages evaluative statements
Ensures that reviewers address all review criteria
and considerations
36
The NIH Peer Review Process
Templates for Reviewers
Links to
definitions
of review
criteria
http://grants.nih.gov/
grants/peer/reviewer_guideline
htm#general_guidelines
37
37
The NIH Peer Review Process
Pre-Meeting SRG Procedures
Reviewers
– Examine assignments
– Complete and submit Conflict of Interest and
Confidentiality voucher
– Read applications, prepare written critiques using
formatted template
– Enter preliminary scores into IAR
– Read and consider other critiques and preliminary scores
– Make travel and hotel arrangements
Preliminary scores and critiques may be
due several days or a week in advance!
38
38
The NIH Peer Review Process
SRG Meeting Procedures
Late grant material
– Acceptance at discretion of SRO
• Correcting errors or omissions, e.g.
– Additional letters of support or collaboration
– Corrected budget form pages
– Information that a manuscript has been accepted
• Cannot modify Research Plan, except for new letters
– Cannot be accepted with fewer than 30 days before the
SRG meeting
39
39
The NIH Peer Review Process
SRG Meetings
Agenda
– Call to Order - Chairperson
– Policy and instructions - SRO
– Discuss applications one at a time
– Where feasible:
• In score order
• Cluster NI applications
• Cluster clinical applications
– Score each application after its discussion
– Discuss other considerations
Remember the
• Budget
Mock Study Section
tomorrow!
• Resource Sharing Plans
40
40
The NIH Peer Review Process
Clustering of New Investigator Applications
• New Investigator (NI):
A PD/PI who has not yet competed successfully for a substantial
NIH research grant
• Early Stage Investigator (ESI):
An NI who is within 10 years of completing the terminal research
degree or medical residency (or equivalent)
41
41
The NIH Peer Review Process
Clustering of New Investigator Applications
New Investigator Policy
– Support New Investigators at success rates equivalent to
that of established investigators submitting new
applications
– Where feasible, NI applications are clustered in review
– NI and ESI applications are identified for reviewers
– Expectations for preliminary data or track record should
– not be the same as for established investigators
42
42
The NIH Peer Review Process
SRG Meeting Procedures
Discussion format
– Members with conflicts excused
– Initial levels of enthusiasm stated
(assigned reviewers and discussants)
– Primary reviewer - explains project, strengths, weaknesses
– Other assigned reviewers and discussants follow
– Open discussion (full panel)
– Levels of enthusiasm (assigned reviewers) re-stated
– Individual SRG members vote
– Other review considerations discussed (budget)
43
43
The NIH Peer Review Process
SRG Meeting Procedures
Reviewer workload
~ 6 – 8 as “reviewer”
~ 2 – 3 as “discussant”
Don’t assume reviewers will
– See the unstated
– Grasp nuances
– Understand your lingo
– Look things up
– Read your mind!
44
44
The NIH Peer Review Process
SRG Meeting Procedures
• If 60 applications/SRG meeting
~ 50% streamlined
30 applications to discuss and score
• If
9 hour SRG meeting
~ ½ hour introduction, streamlining
~ 1 hour lunch, 2 x 15 minute breaks
• Leaves
~ 14 minutes/application
~ 3 - 4 minutes/reviewer
Clarity and brevity are essential!
45
45
The NIH Peer Review Process
Alternate Styles of Review
• Teleconferences
• Editorial-style review
• Video-enhanced discussions
• Internet Assisted Meetings
46
46
The NIH Peer Review Process
After the Review
• NIH Program Officer = Point of Contact
• Wait for summary statement
• Read summary statement carefully
before calling!
A favorable score does not
guarantee funding!
47
47
The NIH Peer Review Process
After the Review
eRA Commons (http://era.nih.gov/commons/index.cfm)
– Final Impact/Priority Score available three days after the
SRG meeting
– Summary statement available 4 – 8 weeks after meeting
• Available to Program Officers at that time
• Confidential document
• Available to:
– PD/PIs
– NIH officials
– Advisory Council members
48
48
The NIH Peer Review Process
Summary Statement
First page
– NIH Program Officer (upper left corner)
• Name
• Contact information
– Final Impact/Priority Score or ND
– Percentile (if applicable)
– Codes
• Human subjects
• Vertebrate animals
• Inclusion plans
– Budget request
49
49
The NIH Peer Review Process
Summary Statement - continued
Subsequent Pages
– Description (provided by applicant)
– Resumé and Summary of Discussion (if discussed)
– Reviewer critiques – essentially unedited
• Follow review criteria for mechanism
• Protections for Human Subjects
• Inclusion Plans
• Vertebrate Animals
• Biohazards
• Budget
– Administrative Notes
50
50
The NIH Peer Review Process
Appeals Process
Consider your options if the outcome is unfavorable
– Revise and resubmit application
• Consider critiques in summary statement
• Address critiques in introduction and text
– Appeal review outcome
• Procedural deficiencies
• Factual errors
• May result in re-review of same
application by different SRG
Discuss your options with your Program Officer!
51
51
The NIH Peer Review Process
Appeals Process
Appeals are submitted to the NIH Program Officer
• All correspondence must be in writing
• All correspondence from the PO must be in hard copy
• Must address procedural flaws in the review, not differences
of scientific opinion
• Can be submitted up to thirty days after Council
• Results of a re-review cannot be appealed further
52
52
The NIH Peer Review Process
Advisory Council/Board
Second level of review
– Make recommendations to IC Director:
• Research priority areas
• Policy
• Appeals
• Funding
• Quality of SRG review
– Concur with SRG recommendations
– Modify SRG recommendations
– Deferral for re-review
– Cannot change final impact/priority score from SRG
–
53
53
The NIH Peer Review Process
Advisory Council/Board
Members
– Scientists from the extramural research community
– Public representatives
– Appointed to multi-year terms
– Appointed as Special Government Employees
– Subject to Standards of Ethics for federal employees
• Rosters: http://www1.od.nih.gov/cmo/committee/index.html
• Schedule: http://www1.od.nih.gov/cmo/committee/index.html
54
54
The NIH Peer Review Process
Advisory Council/Board
Committee
– Subject to the Federal Advisory Committee Act (FACA)
– Balanced representation
• Expertise
• Stature in field
• Mature judgment
• Impartiality
• Managed conflicts of interest
• Balanced representation
– Gender, Diversity
– Geography, Seniority
55
55
The NIH Peer Review Process
Funding Considerations
Funding decisions – IC Director
– Scientific and technical merit
(initial peer review)
– Council recommendations
– Relevance to program priorities in IC
– Number of meritorious applications
received
– Availability of funds
– Advise of IC Program Staff
56
56
The NIH Peer Review Process
Additional Information
• Enhancing Peer Review Initiative
http://enhancing-peer-review.nih.gov/
• Office of Extramural Research Peer Review Process
http://grants.nih.gov/grants/peer_review_process.htm
• Peer Review Policies & Practices
http://grants.nih.gov/grants/peer/peer.htm
• Center for Scientific Review
http://cms.csr.nih.gov/AboutCSR/Welcome+to+CSR/
57
57
The NIH Peer Review Process
Contact Information
Sally Amero, Ph.D.
NIH Review Policy Officer
Extramural Research Integrity Liaison Officer
Office of Extramural Programs
Office of Extramural Research
National Institutes of Health
[email protected]
58
58