Transcript Leslie Hushka 'Peer Review: Challenges Raised by OMB’s

Peer Review:
Challenges Raised by
OMB’s Draft Guidelines
Leslie Hushka, Ph.D., DABT
September 30, 2003
ExxonMobil Biomedical Sciences, Inc.
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
2
What Guideline Elements Are Addressed
 Bulletin
Goals
 Information
 Selecting
Covered / Not Covered
Peer Reviewers
 Peer Review
Process
 Information Access
 Updating Agency
and Public Comments
Guidelines
 Summary
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
3
What is Not Covered in This Section
SRA Forum on Peer Review - May 29 2002
Dr. Richard Belzer, President
Regulatory Checkbook
Dr. Steven Lewis, Distinguished
Scientific Associate, EMBSI

Scholarly peer review  Government
peer review (including ownership,
objectives, selection, procedures,
interests, accountability, etc)
 Peer Review  Stakeholder Dialogue

No evidence that peer review is an
appropriate remedy for the "problem"
government peer review is supposed to
solve

Conflict of Interest  Bias

Too much attention is devoted to
conflicts of interest and not enough to
coincidence of interest

Peer Review  Sound science
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
4
Bulletin Goals
... Ensure that agencies conduct peer reviews of the most important
scientific & technical information
... Peer reviews are reliable, independent and transparent
Questions:
 Are these the proper goals? Are these aligned with info quality?
 What
is reliable?
 Independent?
 What
From Agency only? Transparent how & when?
criteria will be used to judge success in meeting goals?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
5
Information that is Covered / Not Covered by Guidelines
... Regulatory information means any scientific or technical study that is
relevant to regulatory policy . . . used by regulatory bodies
... Peer review undertaken by a scientific journal may be generally be
presumed to be adequate
Questions:
 Does journal peer review meet the amended PR - IQG standards?
 What
if the Agency re-analyzes, summarizes, interpretes original
findings?
 Does
establishing a rebuttable presumption for published articles free
agencies from requirement to substantiate the quality of information?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
6
Selecting Peer Reviewers
... Selected on the basis of necessary scientific and technical expertise
... Broad a range of expertise as is necessary
... Do not possess real or perceived conflicts on interest
Questions:
 Will this create additional burden or discourage qualified scientists?
 Is
a broad range of expertise really necessary? Does it dilute skills &
decrease rigor of review?
 Conflicts
of interest and bias - Does the 'appearance' of either of these
affect selection of reviewers?
 How
will the agency measure 'perceived' conflicts of interest?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
7
Peer Review Process
...Select peer review mechanism based on the novelty and complexity
of the science to be reviewed, the benefits and cost implications, and
any ’controversy’ regarding the science
Questions:
 Are 'controversies' on the science a valid criteria for selecting a peer
review method? Is ‘uncertainty’ a more appropriate term?
 Agency
must provide reviewers with information on those potential
sources of controversy . . .What and how?
 If Agency
uses other firm(s) to conduct review, does it establish some
dependence upon the Agency?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
8
Information Access and Public Comments
...Provide an opportunity for other interested agencies and persons to
submit comments ... provided to peer reviewers
....Disclose names, qualifications of peer reviewers
...Include a certification explaining how agency has complied
Questions:
 How will the Agency address public & peer review comments on
information products that have not been 'disseminated'?
 How
will agencies set up an administrative records for information
products that are years away from use in rulemakings?
 What
criteria will OMB use to review ‘certifications’ for PR/IQG?
Should these criteria be the same to evaluate if a review is successful?
Can criteria be outcome-based and not output-focused?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
9
Updated Agency Guidelines
...Supplement or amend IQGs to incorporate requirements
...Guidelines for... entanglements that preclude an individuals
... Confidentiality in peer review
Questions:
 Performance standards or detailed processes? Is this appropriate?
How will performance be measured?
 Should
separate agency guidelines / standards be developed by each
agency?
 Any
criteria for excluding peer reviewers?
 What,
if anything, is confidential in the peer review process?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
10
Data “Quality” Contains Multiple Elements
Objectivity

Present info in accurate, clear, complete and
unbiased manner
 Identify sources, supporting data and models
 Ensure transparency, reproducibility, and use
“best available science”
Utility
 Agency must consider how the
public uses the information and
ensure appropriate transparency
for those uses
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
Integrity
 Ensure that information is not
compromised through corruption
or falsification
ExxonMobil Biomedical Sciences, Inc.
11
Summary
Many questions for agencies to address in developing or modifying peer
review programs
Overriding questions:
 Will peer review improve the quality of information disseminated?

If so, will peer review also improve the underlying quality of science
used for regulatory purposes?
 If
not, what will?
 Are
there other models that will better fit these needs?
(LJH) SRA / FDA Peer Review Workshop - 9/30/03
ExxonMobil Biomedical Sciences, Inc.
12