Transcript Document

Challenges and Opportunities in Peer Review
A Vision for Ensuring Its Strategic National Value
toni scarpa
[email protected]
301-435-1109
FASEB Board
Bethesda, MD
May 31, 2009
National Institutes of Health
U.S. Department of Health and Human Services
NIH Peer Review
 NIH and Peer Review
 The Drivers for Change
 Enhancing Peer review: CSR
 Enhancing Peer review: Corporate NIH
 ARRA, The Stimulus
NIH Peer Review
30 Years of Medical Innovation
•
•
•
•
•
•
•
•
•
MRI and CT Imaging
ACE inhibitors
Angioplasty
Statins
Mammography
Coronary Interventions
H inhibitors and H2 Blockers
Antidepressant
Cataract and Lens
Replacement
• Knee and Hip Replacement
• Ultrasound Imaging
• Asthma Treatment
•
•
•
•
•
•
•
•
•
•
•
Cardiac Enzymes
Fluoroquinolones
Hypoglycemic Agents
HIV Testing and
Intervention
Tamoxifen
PSA
H. Pylori Test and
Treatment
Bone Densitometry
Cephalosporins
Calcium Blockers
Conscious Sedation
Fuchs and Sox, Health Affairs, 20, 30-42
Why Has The U.S. Biomedical-Behavioral
Research Been So Successful?
• Evolution of unique dynamic partnerships -- through
NIH -- between Government and academic/medical
schools
• 100% of NIH funds to universities and medical
centers awarded through peer review (Only 5-20% in
Europe)
The Overarching Influence of NIH Peer
Review in the USA
For U.S. Universities and Academic Medical Centers
• The promotion committee of medical schools
• The prestige and finances of universities and medical schools
For People and for Public Health
• Which research is done
• Which cures people get
7000 Diseases Affect Humankind
6000 are Orphan Diseases
CSR Peer Review: 2008
• 77,000 applications received
• 16,000 reviewers
• 1,600 review meetings
• 240 Scientific Review Officers
•2,5
CSR Peer Review: 2009
115,000
• 77,000 applications received
• 16,000 reviewers 38,000
• 1,600 review meetings 2,500
• 240 Scientific Review Officers
The Drivers for Change
$ In Billions
$10
$15
$20
$5
$0
Doubling
$25
$29.5
$29.5
$29.1
$28.6
$28.6
$28.0
$27.1
$23.3
$20.5
$17.8
$15.6
$13.7
1st Driver: The NIH Budget
$30
2nd Driver: Number of Applications
80000
60000
Historical Growth
40000
20000
0
1998
2000
2002
2004
2006
2008
3rd Driver: Reviewer’s Load
Applications
Per Reviewer
12
10
8
6
4
2
19
95
19
96
19
97
19
98
19
99
20
00
20
01
20
02
20
03
20
04
20
05
20
06
20
07
20
08
0
4th Driver: CSR Budget
70
$ Millions
60
50
40
30
20
10
0
2004
2005
2006
CSR
2007
Reviewer Cost
2008
Constant $
2009
Annual Savings in Reviewers’ Expense Budget
• Non-refundable tickets with one possible change
$15 million
• 3,000 fewer reviewers
$3 million
• 15% reviews using electronic platforms
$5 million
• One meeting a year on the West Coast
$1.8 million
CSR’s Efforts to Enhance Peer Review
Major Complaints About NIH Peer Review
• The process is too slow
• There are not enough senior/experienced reviewers
• The process favors predictable research instead of
significant, innovative, or transformative research
• The time and effort required to write and review are a
heavy burden on applicants and reviewers
CSR’s Efforts to Enhance Peer Review
1. Reorganizing CSR and Recruiting Staff
2. Improving Study Section Alignment
3. Shortening the Review Cycle
4. Advancing Additional Review Platforms
5. Recruiting the Best Reviewers
6. AATS Peer Review Outcome
1. Reorganizing CSR
Neuroscience,
Development
and Aging
AIDS, Behavioral,
Population
Brain Disorders &
Clinical Neuroscience
Biobehavioral &
Behavioral Processes
Molecular, Cellular &
Developmental Neuroscience
Risk, Prevention&
Health Behavior
Integrative, Functional &
Cognitive Neuroscience
Emerging Technologies &
Training in Neuroscience
Biology of Development &
Aging
Basic- Integrative
Biological Sci
Physiological
Pathological Sci
Translational and
Clinical Sci
Biological Chemistry &
Macromolecular
Biophysics
Endocrinology,
Metabolism, Nutrition &
Reproductive Sciences
Cardiovascular and
Respiratory Sciences
Bioengineering Sciences
& Technologies
Immunology
Surgical Sciences,
Biomedical
Imaging and
Bioengineering
Cell Biology
Infectious Diseases
& Microbiology
Musculoskeletal, Oral
And Skin Sciences
Digestive, Kidney &
Urological Systems
Oncology 2 –
Translational Clinical
Population Sciences
and Epidemiology
Healthcare Delivery
& Methodologies
AIDS &
Related Research
Genes, Genomes
&Genetics
Oncology 1 – Basic
Translational
Interdisciplinary
Molecular
& Training
Vascular and
Hematology
2. Improving Study Section Alignment
•
Input from the community
•
Internal IRG reviews
•
Open Houses
•
PRAC
3. Shortening the Review Cycle
Why?
First Response was 5.2 months, far too long
The Goal
To review and post score and critique application
within 3 months of submission.
To enable resubmission, when doable and desirable, 4
months earlier than in the past.
The Result
Every New Investigator and Most Established
Investigators are eligible
13% of those eligible apply
4. Advancing Additional Review Platforms
• Additional Review Platforms Help Recruiting
Reviewers
•
Electronic Review Modes Reduce Travel
• Electronic Reviews
Telephone Enhanced Discussions
Video Enhanced Discussions
Asynchronous Electronic Discussions (AED)
4. Advancing Additional Review Platforms
Reviewer Satisfaction with AED Review
5. Recruiting the Best Reviewers
16000
14000
12000
10000
8000
6000
4000
2000
0
1999 2000 2001 2002 2003 2004 2005 2006 2007 2008
Chartered
Temporary
5. Recruiting the Best Reviewers
Academic
Rank
of All
CSR Reviewers
Academic Rank
of ALL
CSR
Reviewers
80.0%
70.0%
60.0%
50.0%
40.0%
30.0%
20.0%
10.0%
0.0%
1998
1999
2000
PROFESSOR
2001
2002
2003
2004
ASSOCIATE PROFESSOR
2005
2006
2007
ASSISTANT PROFESSOR
2008
5. Recruiting the Best Reviewers
Some Successful Strategies
•Move a meeting a year to the West Coast
•Additional review platforms
•Develop a national registry of volunteer reviewers
•Searchable database with 5,000 reviewers
•Provide tangible rewards for reviewers
•No submission deadlines for chartered members
• of study sections (effective February 2008).
•Provide flexible time for reviewers
•Choice of 3 times/year for 4 years or
•2 times/year for 6 years
Enhancing Peer Review
The NIH Director’s Recommendations
Corporate NIH: Enhancing Peer Review
• The Charge from Dr. Zerhouni:
“Fund the best science, by the best scientists,
with the least administrative burden…”
Two advisory committees to the NIH Director
http://enhancing-peer-review.nih.gov
The Process
Diagnostic
Design
Implementation
Plan
June 2007 – Feb. 2008
March 2008 – June 2008
Begin Phased
Implementation
of Selected
Actions
September 2008
2
1. Early Stage Career Investigators
Definition of New Investigator:
•
Not previously competed successfully as PD/PI for a significant
NIH independent research award.
Definition of Early Stage Investigator:
•
Within 10 years of completing terminal research degree or within
10 years of completing medical residency (or the equivalent).
• The NIH corporate policy is to fund R01s of New
Investigators and ESIs at different paylines
•
• 3 Paylines for R01s Applies only to R01 applications
• Advice for ESI (and to New PI)
Apply NOW!!!
Apply for R01!!!!!
2. Review Highly Transformative Research
• OD Transformative RO1 (T-RO1)
Once a year, 5 years, $40 million total budget
Deadline January 29, 2009
8-page application
740
Editorial Board Review
o Heavy triage based on innovation and potential science
transformation by a small study section of distinguished,
broad-science reviewers (the editors)
o Specific science reviewed by appropriate reviewers (the
editorial board)
o Final ranking by the editors
3. Funding the Best Research Earlier
• More flexible deadlines
• Abolish A2 applications
4. Enhanced Review
• Overall Impact:
Assessment of the likelihood for the project to exert
a sustained, powerful influence on the research
field(s)
• New Core Criteria Order:
Significance
Investigator(s)
Innovation
Approach
Environment
4. Template-Based Critiques
• The objective is to write evaluative statements and to avoid
summarizing the application
• Comments should be in the form of bullet points or if
necessary short narratives
• The entire template is uploaded to IAR to become part of
the summary statement.
1. Significance
Strengths



Weaknesses



Please limit text to ¼ page
4. Scoring
Impact
High Impact
Moderate Impact
Low Impact
Score
Descriptor
1
Exceptional
2
Outstanding
3
Excellent
4
Very Good
5
Good
6
Satisfactory
7
Fair
8
Marginal
9
Poor
4. The Essence of Scoring
• Before the Meeting
• Every assigned reviewer will post criteria scores AND overall
impact scores.
• The impact scores will be used to determine the order of
discussion.
• During the Meeting
• For the discussed applications:
The overall impact score is stated by the assigned reviewers.
Criteria scores are not mentioned during the discussion.
• After the Meeting
• The applicant with a discussed application will see 3 or more
criteria scores AND the overall impact score (the one that will
be percentiled)
• The applicants with a non discussed application will see ONLY
the criteria scores (3 or more for each criterion)
4. Enhancing Peer Review Training
•
CSR and NIH Review Staff
6 face to face training sessions, January 2009
6 face to face training sessions, April 2009
Continuous updating
•
Chairs
For Chairs appointed in 2008, 8 sessions in January 2009
For Chairs appointed in 2009, 7 sessions in April-May 2009
•
Reviewers
Training material (Power Point, interactive training, frequently asked
questions, mock study section video, etc,in April-May 2009
Senior CSR staff at the first meeting in May-July 2009
American Recovery and
Reconstruction Act
The Stimulus
ARRA Applications Under Review
• RC1. Challenge Grants
Deadline April 27
Received so far
Verified and Assigned to IRGs
Assigned to 3 Reviewers
20,847
19,107
14,313
2 Stages Editorial Board Review
o Each application assigned to 3 reviewers (20,000), reviews
due by June 5th, extended to June 12th
o 30 Special Study Sections in early July
• Competitive Revisions
Deadline April 23
Received 1,985
Large Majority Assigned to the Standard 250 Study Sections
Study Sections will review as a SEP in May-June
Stem Cells Challenges
This is CSR
September 2008