Transcript Document

CSR’s Mission and Function
and
What’s New in Peer Review
Martha M. Faraday, Ph.D.
Scientific Review Officer
Division of AIDS, Behavioral & Population Sciences
Risk Prevention & Health Behavior IRG
Psychosocial Risk & Disease Prevention Study Section
Date : April 22, 2009
National Institutes of Health
U.S. Department of Health and Human Services
1
National Institutes of Health
Office of the Director
National Institute
on Aging
National Institute
on Alcohol Abuse
and Alcoholism
National Institute
of Allergy and
Infectious Diseases
National Institute
of Arthritis and
Musculoskeletal
and Skin Diseases
National Cancer
Institute
National Institute
of Child Health
and Human
Development
National Institute on
Deafness and Other
Communication
Disorders
National Institute
of Dental and
Craniofacial
Research
National Institute
of Diabetes and
Digestive and
Kidney Diseases
National Institute
on Drug Abuse
National Institute
of Environmental
Health Sciences
National Eye
Institute
National Institute
of General
Medical Sciences
National Heart,
Lung, and Blood
Institute
National Human
Genome Research
Institute
National Institute
of Mental Health
National Institute
of Neurological
Disorders and
Stroke
National Institute
of Nursing Research
National Institute of
Biomedical Imaging
and Bioengineering
National Center
for Complementary
and Alternative
Medicine
Fogarty
International
Center
National Center
for Research
Resources
National Library
of Medicine
National Center on
Minority Health and
Health Disparities
Clinical Center
Center for
Information
Technology
Center for
Scientific Review
CSR Mission Statement
To see that NIH grant applications receive
fair, independent, expert, and timely reviews free from inappropriate influences - so the
Institutes and Centers within the NIH can fund
the most promising research.
CSR Peer Review: 2008 Statistics
• 77,000 applications received
• 56,000 applications reviewed
• 16,000 reviewers
• 240 Scientific Review Officers
• 1,600 review meetings
Scientific Review Process
Dual Review System for Grant Applications
First Level of Review : CSR/Institute Review
Scientific Review Group (SRG)
(Study Section)
Second Level of Review
NIH Institute/Center Council
CSR Review Divisions
AIDS, Behavioral
Neuroscience, Development
and Population Sciences
and Aging
Brain Disorders &
Clinical Neuroscience
Molecular, Cellular &
Developmental Neuroscience
Integrative, Functional &
Cognitive Neuroscience
Emerging Technologies &
Training in Neuroscience
Biology of Development
& Aging
Biobehavioral &
Behavioral Processes
Risk, Prevention&
Health Behaviors
Population Sciences &
Epidemiology
Healthcare Delivery
& Methodologies
AIDS &
Related Research
Basic and Integrative
Physiological and
Biological Sciences Pathological Sciences
Biological Chemistry &
Macromolecular
Biophysics
Bioengineering Sciences
& Technologies
Cell Biology
Genes, Genomes
& Genetics
Oncology 1 – Basic
Translational
Interdisciplinary
Molecular
& Training
Translational and
Clinical Sciences
Endocrinology,
Metabolism,
Nutrition &
Reproductive Sciences
Cardiovascular and
Respiratory Sciences
Immunology
Surgical Sciences,
Biomedical
Imaging and
Bioengineering
Infectious Diseases
& Microbiology
Musculoskeletal, Oral
And Skin Sciences
Digestive, Kidney &
Urological Systems
Oncology 2 –
Translational Clinical
Vascular and
Hematology
Assignment to CSR Review Groups
Within an IRG, applications are assigned
for review to
• Standing Study Sections when the subject matter
of the application matches the referral guidelines
for the study section
• Special Emphasis Panels (SEPs) when the subject
matter does not fit into any study section, or when
assignment of an application to the most
appropriate study section would create a conflict
of interest. Also used for special mechanisms
(e.g., fellowships, SBIRs, AREAs)
When Preparing Your Application
• Read the instructions
• Never assume that reviewers “will know what you
mean”
• Refer to the literature thoroughly
• State rationale of proposed investigation
• Include well-designed tables and figures
• Present an organized, lucid write-up
• Remember to address human subjects, vertebrate
animals, potential biohazards; these could affect
your score
• Obtain pre-review from faculty at your institution
NIH Grant Writing Tips:
http://grants.nih.gov/grants/grant_tips.htm
Directing Your Application to a Specific
Study Section
• Peruse CSR Study Section Guidelines* to Identify
a Possible Home for Your Application
http://csr.nih.gov/
* Recently revised; alternative study sections
listed in approximate order of degree of overlap
• Submit a Cover Letter
CSR Web Site: http://www.csr.nih.gov
• About CSR
• News and Reports
• Peer Review Meetings
• Resources for Applicants
Study Section
Descriptions & Rosters
Role of Scientific Review Officer (SRO)
Designated Federal official with overall
responsibility for the review process
• Performs administrative and technical review of
applications to ensure completeness and accuracy
• Selects reviewers based on broad input
• Manages study section meetings
• Prepares summary statements
• Provides any requested information about study section
recommendations to Institutes/Centers and National
Advisory Councils/Boards
WHOM DO I CONTACT?
 Before review, contact the Scientific Review
Officer in CSR
 After review, contact your Program Officer in
the NIH funding institute or center
Pre-Meeting Review Process
• Appropriate reviewers recruited by SRO;
minimum of 3 “interactive” reviewers per
application
• Conflicts of interest identified
• Applications made available to reviewers ~6
weeks prior to meeting
• Critiques and preliminary scores posted by
assigned reviewers on NIH web site at least 2-3
days prior to meeting
• Critiques and preliminary scores (excluding
conflicts) available to review group prior to
meeting
Where Do We Find Reviewers?
• National Registry for SocietyRecommended Reviewers
• Successful applicants
• Word of mouth
Recommendations from study section
members
Recommendation from NIH IC staff
• CRISP (crisp.cit.nih.gov)
• PubMed
• Scientific Conferences
Traditional* Review Meeting Process
• Upper half applications discussed:
Reviewers are guided by specific review criteria
Protections for Humans, Vertebrate Animals,
Environment (Biohazard) may affect final score
Assigned reviewers recommend scores for each
application in upper half; all members not in conflict vote
their conscience (outlier score policy pertains)
Other considerations not affecting final score are
discussed (e.g., budget, foreign applicants, resource
sharing plans)
• Lower half applications not discussed, not assigned an
overall score
* Aspects of this process will change in May, 2009
http://enhancing-peer-review.nih.gov
Post Meeting Review Process
•
Scores are provided to investigators within 3 working days
•
Summary Statements for discussed and scored applications
include Resume & Summary of Discussion, (largely unedited)
critiques, and other recommendations (e.g., Budget)
•
Summary Statements for lower half (Not Discussed)
applications receive (largely unedited) critiques and review
criteria scores but no overall impact scores
•
All Summary Statements are made available within 30 days of
meeting (10 days for new investigators’ R01s)
What’s New in Peer Review?
2008: The Year of Peer Review
Enhancing Peer Review
“Fund the best science, by the best scientists,
with the least administrative burden…”
Elias Zerhouni, MD, Former Director, NIH
Recommendations
Priority 1:
Engage the Best
Reviewers
Increase Flexibility to Better
Accommodate Reviewers
Recruit Reviewers
Acknowledge Reviewers
more formally
Compensate Reviewers
Time and Effort
Improve Review Quality with
Training
Priority 2:
Improve the
Quality &
Transparency of
Review
Modify Rating System to
Focus on Specific Review
Criteria
Align Summary Statement
with Review Criteria
Shorten and Align
Application with Review
Criteria
Priority 4:
Continuous
Review of
Peer Review
Priority 3:
Ensure Balanced &
Fair Reviews
Across Scientific
Fields and Career
Stages
Support for Early Stage
Investigators
Review of Established
Investigators
Enhanced Review of Clinical
Research
Expand Awards Encouraging
“Transformative Research”
Reduce Need for
Resubumissions
Amended Applications:
To speed the funding of meritorious science and
minimize reviewer burden:
•
As of January 25, 2009, all original new applications (i.e.,
never submitted) and competing renewal applications will
be permitted only a single amendment (A1).
What’s New in Peer Review
• New Investigators/Early Stage Investigators
• Enhanced Review Criteria
• Template-Based Critiques
• Scoring Scale (9 point scale)
Criterion Scoring
Overall Impact Score
21
New Investigators/Early Stage Investigators
• New Investigator (NI):
PD/PI who has not yet competed successfully for a
substantial NIH research grant
o For multiple PD/PIs-all PD/PIs must meet requirements for NI status
• Early Stage Investigator (ESI):
PD/PI who qualifies as a New Investigator AND is within 10
years of completing the terminal research degree or is within
10 years of completing medical residency (or equivalent)
•
•
Applies only to R01 applications
New Investigators/Early Stage Investigators will be clustered
together for review
22
Enhanced Review Criteria
• Overall Impact:
Assessment of the likelihood for the project to
exert a sustained, powerful influence on the
research field(s) involved
• New Core Criteria Order:
Significance
Investigator(s)
Innovation
Approach
Environment
o Review criteria enhanced and expanded
23
Template-Based Critiques
Critiques
Goal: To improve the quality of the critiques and to
focus reviewer attention on the review criteria:
• Provide clear, concise, and explicit information
• Aid in identifying the strengths and weaknesses
of each criterion
Template-Based Critiques
• Critique template contains a total of 18 boxes
Reviewers should provide text for only those criteria that
are applicable.
1. Significance
6. Resubmission
13. Overall Impact
2. Investigator(s)
7. Renewal
14. Budget and Period of
Support
3. Innovation
8. Revision
15. Select Agents
4. Approach
9. Protection of Human
Subjects
16. Applications from
Foreign Organization
5. Environment
10. Inclusion of Women,
Minorities, and Children
17. Resource Sharing
Plan
11 Vertebrate Animals
18. Additional
Comments to
Applicant
12. Biohazards
25
Template-Based Critiques
• Goal: is to write evaluative statements and to
discourage summarizing the application
Comments should be in the form of bullet points or if
necessary short narratives
Do not record scores on the critique template
The entire template is uploaded to IAR to become part of
the summary statement.
1.
Significance
Please limit text to ¼ page
Strengths



Weaknesses



26
Scoring – 9 Point Scale
Goal: To improve the transparency of the scoring
process:
• Score applications on five review criteria using
a scale of 1-9.
• Preliminary overall impact score using 1-9
scale.
Should not be the average of the criterion
scores.
Not Discussed applications will receive initial
criterion scores from the three assigned
reviewers
Scoring Descriptions
Impact
High Impact
Moderate Impact
Low Impact
Score Descriptor
1
Exceptional
2
Outstanding
3
Excellent
4
Very Good
5
Good
6
Satisfactory
7
Fair
8
Marginal
9
Poor
Strengths/Weaknesses
Strengths
Weaknesses
Clustering
• NI/ESI R01 applications will be clustered
together in review.
ESI applications will not be separately
clustered within the NI\ESI group.
o NI/ESI applications will be identified for
reviewers so there can be appropriate review
in context of career stage.
o Expectations of preliminary data and
publication track record less than for
established investigators.
Order of Review
Goal: Discuss applications in order of average
preliminary score.
Why:
•Concern - variation of scores during different times
of the meeting.
One recommendation was to recalibrate scores at
the end of the meeting .
Solution:
•Recalibrate “dynamically” throughout meeting.
Order of Review
• For calibration purposes:
Begin meeting by discussing the best scored
application (any activity code)
o NI/ESI R01s clustered beginning of meeting
o All other activity codes clustered if feasible (if at
least 10 discussed (may include R03, R15, and
R21s as a group that can be clustered)
Order of Review
Summary
Discussion order is based on the average of
the impact scores from assigned reviewers
Final scores of discussed applications may
differ from preliminary scores as recalibration happens dynamically
Not Discussed
Discuss ~ 50-60% of applications
SRO will then ask if there are any other
applications that panel wishes to discuss
The remaining applications will not be discussed
(applications receive criterion scores only)
o Same after review of ~60% of SBIR applications
Final Scores
• Discussed applications will receive an overall score
from each eligible (i.e., without conflicts of interest)
panel member and these scores will be averaged to
one decimal place, and multiplied by 10. The 81
possible priority scores will thus range from 10-90.
• Percentiles will be reported in whole numbers.
Summary Statements
• Summary statement will be shorter and more
focused.
• Discussed applications will also have a summary
of the panel’s discussion at the meeting.
• ALL applications will be scored.
Not discussed applications will receive
criterion scores only.
Recruiting the Best Reviewers
 Move a meeting a year to the West Coast
 Additional review platforms
 Develop a national registry of volunteer reviewers
 Searchable database with 4,000 reviewers
 Provide tangible rewards for reviewers
 No submission deadlines for chartered members
of study sections (effective February 2008).
 1574 chartered members used flexible deadlines
during the last 6 months
Provide flexible time for reviewers
 Choice of 3 times/year for 4 years or
2 times/year for 6 years
THANK YOU!
This is CSR