The New NIH Review System: Reviewer’s perspective

Liz Madigan, FPB School of Nursing

• NIH information/guidance • Advantages/disadvantages as a reviewer • Recommendations for further improvement

8,0 7,0 6,0 5,0 4,0 3,0 2,0 1,0 1,0 9,0

Relationship of Old vs. New Scores

Old vs. New Scores

y = 3,2357x - 2,1153 R² = 0,7617 1,5 2,0 2,5 3,0

Old

3,5 4,0 4,5 5,0

NIH Guidance

• • The NIH grant application scoring system uses a 9-point rating for the impact/priority score Assigned reviewers also provide ratings for each review criterion (Significance, Investigator(s), Innovation, Approach, Environment) using the same 9-point scale. – These criterion ratings are provided in the summary statement of all applications, both discussed and undiscussed.

– Criterion ratings should be considered in determining the overall impact/priority score, but because the relative importance of each criterion to the overall impact/priority score differs for each application, reviewers should not apply a formula of unweighted or weighted criterion scores across applications.

NIH Guidance

• Reviewers are strongly encouraged to utilize the full range of the rating scale in determining ratings. Optimally, scores will be normally distributed with very few 1’s and 9’s and a majority of scores in the middle of the range (4-6). • Discussed applications will receive impact/priority scores from all eligible (not in conflict) reviewers, and these scores will be averaged and multiplied by 10 to determine the final impact priority score (range of 10 to 90).

• Because the relative importance of each individual criterion to the overall score differs for each application, reviewers should not use a formula of weighted or unweighted averages across applications to determine the overall impact/priority score.

Score 1 2 3 8 9 4 5 6 7 Descriptor Exceptional Outstanding Excellent Very Good Good Satisfactory Fair Marginal Poor Additional Guidance on Strengths/Weaknesses

Exceptionally strong with essentially no weaknesses Extremely strong with negligible weaknesses Very strong with only some minor weaknesses Strong but with numerous minor weaknesses Strong but with at least one moderate weakness Some strengths but also some moderate weaknesses Some strengths but with at least one major weakness A few strengths and a few major weaknesses Very few strengths and numerous major weaknesses Minor:

An easily addressable weakness that does not substantially lessen the impact the project

Moderate:

A weakness that lessens the impact of the project

Major:

A weakness that is severely limits the impact of the project

2 3 4 5 6 7 8 9 Score 1 Descriptor Exceptional Outstanding Excellent Very Good Good Satisfactory Fair Marginal Poor Significance Descriptors

Achieving the proposed aims is likely to advance the research field in profound and lasting ways Achieving the proposed aims is likely to advance the research field in critically important ways Achieving the proposed aims is likely to advance the research field in important ways Achieving the proposed aims is likely to contribute substantially to the current knowledge base of the research field Achieving the proposed aims is likely to contribute meaningfully to the current knowledge base of the research field Achieving the proposed aims is likely to contribute somewhat to the current knowledge base of the research field Achieving the proposed aims is likely to contribute only incrementally to the current knowledge base of the research field Achieving the proposed aims is likely to contribute only minimally to the current knowledge base of the research field Achieving the proposed aims is unlikely to contribute in any way to the current knowledge base of the research field

8 9 4 5 6 7 Score 1 2 3 Descriptor Exceptional Outstanding Excellent Very Good Good Satisfactory Fair Marginal Poor Investigator Descriptors

The investigators are extremely well qualified to achieve the proposed aims.

The investigators are qualified to achieve the proposed aims.

The investigators do not appear to have adequate qualifications to achieve the proposed aims.

NIH-provided Word Template

• Specified template provided that reviewers were to use • “Limit text to ¼ page” for each of the criterion areas – Significance – Investigators – Innovation – Approach – Environment

Reviewer Advantages

• Broader range of scores and more descriptions made it somewhat easier to rank applications within each criterion • Identifying strengths and weaknesses made the review more focused • Review was easier to write in some ways —bullet points of strengths and weaknesses

Reviewer Disadvantages

• Impact/priority score was still difficult for some applications (e.g. very experienced team, well funded in the past, application extended the work somewhat but not in a very exciting way OR application from relatively new investigator that may move field forward but approach was not precise or defined) • Worries about the scope and extent of comments being given with the suggested limitations in the review length

Reviewer Recommendations

• Need to indicate for the applicants which weaknesses are minor, moderate and major so the applicants can revise accordingly —we were not forced to do this in the review so I worry that the applicants are getting unprioritized comments • Recommendations for ¼ page of text is difficult and may not result in higher quality reviews, applications and revisions