Transcript peer review

NO CONFLICT OF INTEREST
TO DECLARE
Can peer review improve
quality?
David Smith
Peer review origin?
Ethics of the Physician
Ishaq bin Ali al-Rahwi (854–931) of Syria
All Quality Assurance programmes seem
to rely ultimately on peer review, often in
the form of an expert panel, to set the
standards and the limits of acceptability.
Problems with peer review
• Provision of
– objective standards
– structured processes
– training reviewers
• Perverse incentives/perverse activities
• Reviewer bias
• Authority of the reviewers
• Indemnity/protection
GMC procedures to assess the performance
of doctors who may be seriously deficient
• peer review of the doctor’s practice at the workplace
• tests of competence and skills.
Peer reviews are conducted by three trained assessors, two
from the same speciality as the doctor being assessed, with
one lay assessor and include:
– the doctor’s portfolio of training, experience, practice and self
assessment
– a review of the doctor’s medical records
– discussion of cases selected from these records
– observation of consultations
– a tour of the doctor’s workplace
– interviews with at least 12 third parties (five nominated by the
doctor)
– structured interviews with the doctor.
Failure of peer review
Given the magnitude of the resources devoted to
quality assurance and the centrality of peer
assessment to these efforts, the results of this
literature analysis indicate the need for a global
reexamination of the peer review process.
Several of the proposals discussed herein appear to
have considerable potential for improving the
reliability of peer judgments.
Research directed at evaluating this potential should
be part of the overall reassessment of peer review.
Goldman RL JAMA. 1992;267:958-960
Reviewer agreement
physician agreement regarding quality
of care is only slightly better than the
level expected by chance.
Goldman RL JAMA. 1992;267:958-960
Goldman suggestions
•
•
•
•
•
Use of multiple reviewers
More objective assessments
Higher standards for peer reviewers
Use of practice guidelines
Use of outcome assessments
Watching the doctor watchers
The screening process also must be more
accurate in order to be cost-effective, as it
was only slightly better than random
sampling at correctly identifying below
standard care. More reproducible physician
review is also needed and might be
accomplished through
– improved reviewer selection and training
– a structured review method
– more physician reviewers per record
Rubin et al JAMA 1992;267:2349-2354
Review of 339 hospitals PR system
• Only 24% of respondents said it was very likely to
improve quality while 33% said it was unlikely
•
•
•
•
Needs standardisation,
more governance,
bigger case load for reviews (>1% of admissions),
more personnel (already 1.1wte per 100 beds)
The Process of Peer Review in U.S. Hospitals
Marc T. Edwards & Evan M. Benjamin
JCOM 2009 Vol. 16;10 461-467
Reviewer agreement
•
•
•
•
•
•
13 board-certified physicians completed a total of 741 structured implicit record
reviews of 95 records for patients who experienced severe in hospital adverse events
while in the hospital (hypokalemia, hyperkalemia, renal failure, hyponatremia, and
digoxin toxicity).
They independently assessed the degree to which each adverse event was caused
by medical care and the quality of the care leading up to the adverse event.
Working in pairs, they then discussed differences of opinion, clarified factual
discrepancies, and re-rated the record. The authors compared the reliability of each
measure before and after discussion, and between and within pairs of reviewers.
The reliability of the assessment of whether the complication was iatrogenic was 0.46
before discussion and 0.71 after discussion between paired reviewers, indicating
considerably improved agreement between the members of a pair.
However, across reviewer pairs, the reviewer reliability was 0.36 before discussion
and 0.40 after discussion.
Similarly, for the rating of overall quality of care, reliability of physician review went
from 0.35 before discussion to 0.58 after discussion as assessed by pair. However,
across pairs the reliability increased only from 0.14 to 0.17
Conclusions. When 2 physicians discuss a record that they are reviewing, it substantially
improves the agreement between those 2 physicians. However, this improvement is
illusory, as discussion does not improve the overall reliability as assessed by
examining the reliability between physicians who were part of different discussions .
Hoper TP et al Medical Care 2000, 38:2 152-161
Guidelines
Surrogate markers of quality
May be ill founded
May be biased
Open to interpretation
Unintended consequences
Limit development
Are guidelines following guidelines?
A review of 279 guidelines
1985-1997
Guidelines published in the peer-reviewed medical
literature during the past decade do not adhere well
to established methodological standards. While all
areas of guideline development need improvement,
greatest improvement is needed in the identification,
evaluation, and synthesis of the scientific evidence.
JAMA. 1999;281(20):1900-1905
Practice guidelines developed by specialty
societies: the need for a critical appraisal
431 guidelines were eligible for the study. Most did not meet the
criteria
Despite improvement over time, the quality of practice guidelines
developed by specialty societies is unsatisfactory.
Explicit methodological criteria for the production of guidelines
shared among public agencies, scientific societies, and patients'
associations need to be set up.
Lancet 2000:355;103-106
Scientific Evidence Underlying the ACC/AHA
Clinical Practice Guidelines
Recommendations issued in current ACC/AHA clinical practice
guidelines are largely developed from lower levels of evidence
or expert opinion.
The proportion of recommendations for which there is no
conclusive evidence is also growing.
These findings highlight the need to improve the process of
writing guidelines and to expand the evidence base from which
clinical practice guidelines are derived
JAMA. 2009;301(8):831-841
Percentage actual and
predicted MACCE outcome
Risk adjusted outcomes
funnel plots
10.00%
9.00%
8.00%
7.00%
6.00%
5.00%
4.00%
3.00%
2.00%
1.00%
0.00%
Actual
Predicted
LCL
UWL
UCL
LWL
0
500
1000
1500
Number of cases
2000
2500
BCIS ‘peer review’ and
angioplasty quality
Historical ‘no holds barred’ case review is
an exemplar of peer review but to be
rigorous it needs a systematic format and
a written record
A prospective study of colonoscopy
practice in the UK today: are we
adequately prepared for national colorectal
cancer screening tomorrow?
C J A Bowles
GUT 2004 53 277-283
Can peer review improve
quality?
Increasingly robust risk adjusted
outcome data
Local systematic egalitarian case
review arrangements
BCIS Peer Review
• What about BCIS per review
• BCIS were asked to set up a PR system in 1999 and felt individual
PR was not feasible and designed a unit PR system.The first draft
had many of the features that Goldman suggests but fell at the first
few hurdles of reviewer acceptability, resources, lack of indemnity
• At that point it was highjacked by the department of Health in order
to limit the potential proliferation of of low grade angioplasty units. It
has continued in this format of a box ticking exercise for new units
measured against current BCIS guidelines.
• limited as it is does it improve quality? One of the few but always
contentious, surrogate measures of quality that are recorded is the
number of procedures done per year.
• Yet there are some units consistently doing less than 400 for up to 7
years
Numbers of small volume units
over last ten years
20
18
16
14
12
<200
200-400
10
8
6
4
2
0
2001
2
3
4
5
6
7
8
9
Report Headings
1.Introduction and outline of visit
1.Background
1.Business Case
1.Cardiac Network support
1.Relationship with other PCI Centres
1.Surgical service support
1.Ambulance service support
1.Facilities and Equipment
1.Staffing and on-call arrangements
1.Protocols for Patient Care
1.Audit and National Returns of Activity and Outcome
1.Commentary and Conclusions
Appendix – specific notes from staff interviews
Number of PCIs performed in 2006
2006 data: Ludman
Centres performing < 200 procedures
St Peter's Hospital, Chertsey (2003)
Whipps Cross (2005)
Southend Hospital (2004)
Lister Hospital (2006)
Craigavon Hospital (2006)
Eastbourne Hospital (restart 2006)
Worthing Hospital (2006)
King Edward VII (closed 2006)
Great Western Hospital, Wiltshire (2006)
Wexham Park Hospital (2006)
Sunderland Royal Hospital (2006)
East Surrey Hospital (2006)
0
PCI program start date
20
40
60
80
100 120 140 160 180 200
• The purpose of peer review is to provide an expert view
of an individual’s ability to meet professionally
acceptable standards and thus help ensure an
acceptable level of continued professional competence.
Such review might necessarily identify individuals who
do not conform to acceptable standards and who may
therefore be required to undergo retraining or be
suspended. If the outcome of a peer review might put at
risk the ability of an individual to continue to practice,
then such a review has to be rigorous in the extreme. It
would need to be supported by a large volume of
accurate statistics and involve comparison to a detailed
set of performance standards that is accepted by the
speciality as a whole
• Audit
• Case review
• Direct observation of clinical
activity/procedures
• 360 Appraisal