Transcript Measuring Health Care Quality

The Goals of Guidance:
Maximizing Guideline Value and Benefit
J. Sanford (Sandy) Schwartz, MD
Leon Hess Professor of Medicine and
Health Management & Economics
Perelman School of Medicine &
Wharton School of Business
University of Pennsylvania
Penn
Extramural Activity Disclosure (12 month)
Consultant
Association of University Radiologists/American College of Radiology
Bayer
Christiana Care
Intermountain Health Care
Mathematica
Pfizer
Research Support
National Institutes of Health (NIA, NCI, NHLBI, NIDDK)
Pfizer
Robert Wood Johnson Foundation
Board of Directors
ABC House of Lower Merion
ECRI Institute
Jewish Social Policy Action Network (JSPAN)
National Scientific Advisory Committee Member/Consultant
BlueCross/BlueShield Associations (Medical Advisory Panel)
Centers for Medicare and Medicaid Services (Medicare Evidence Development and Coverage
Advisory Committee)
Institute of Medicine National Academy of Sciences
National Institutes of Health (Advisory committees clinical practice guidelines, comparative
effectiveness research, value of information analysis)
US Preventive Services Task Force
Penn
Clinical and Policy Guideline Development
Personal Perspective
• American College of Physicians Clinical
Efficacy Assessment Project, Founding Director
• NIH Consensus Development Conferences
• National Heart Attack Alert Program
• Blue Cross/Blue Shield Technology Evaluation
Center Medical Advisory Panel (MAP)
• US Preventive Services Task Force
• CMS Medicare Evidence Development Coverage
Advisory Committee (MEDCAC)
• NHLBI ATP III, ATP IV, Integrated CVD guideline
panels
Penn
Summary
• Guidelines should be patient centered, focusing
on individual patient decision making needs
• While we have made much progress over the
past 30+ years, the evidence base for ‘evidence
based guidelines’ must be improved
– ‘Fit for Purpose – broader meaningful
incorporation of data and information
generated by study designs beyond the
excessive focus on RCTs
– Outcomes measured (preferences, costs) and
how (time horizon, settings, behavior)
– Analytical methods (validated)
Penn
Clinical Practice Guidelines
“ … statements that include recommendations
intended to optimize patient care that are informed
by a systematic review of evidence and an
assessment of the benefits and harms of
alternative care options.”
– Institute of Medicine, 2011
Clinical Practice Guidelines We Can Trust
Penn
Penn
What Will Guidelines Do?
• Convert science-based knowledge to clinical
action and connect outcomes research to
clinical practice
• Allow respected professional judgment to advise
the researcher on what needs to be done
• Clarify medical care choices for the consumer
and make explicit different standards of care
where they exist
• Strengthen link between quality and
management of health care
AHRQ conference presentation, 1994?
Penn
Potential Guideline Targets
• Patients
• Clinicians
• Purchasers/payers
• Policy makers
To informe decisions that will improve health care
at both the individual and population levels
Penn
Population Paradoxes
• Large benefit to population may provide limited
benefit to individuals
• Large benefit to an individual may have a small
population impact
• Modest benefit to an individual may be wiped out
by a small harm to many
• Discounting is counter to behavioral preferences
• Social benefits may not be attractive to the
individual, even if beneficial on a
social/community basis
Penn
Clinical Practice Guidelines:
Patient-Centered Outcomes Research Working Definition
Informs patient–focused questions targeted toward
individual patient needs:
• Given my personal characteristics, conditions and
preferences, what should I expect will happen to
me?
• What are my options and what are the benefits and
harms of those options?
• What can I do to improve the outcomes that are
most important to me?
• How can the health care system improve my
chances of achieving the outcomes I prefer?
Penn
“I have yet to see any problem,
however complicated,
which … looked at it in the right way,
did not become still more complicated”
– Poul Anderson
Penn
“Absence of evidence of effectiveness”
increasingly will be interpreted as
“evidence of absence of effectiveness”
Penn
Clinical Guidelines: Requirements
• Patient centered
• Evidence based
• Outcomes focused
• Cost–effective
Penn
Clinical Guidelines: Challenges
• Information gaps
• Too narrow framing
– Conditions/Risks
– Measures
– Methods
– Outcomes
– Time horizons
Penn
Evaluation of Medical Care
Safety
Efficacy
Effectiveness
Side effects acceptable?
Can it work?
Does it work?
Balance of benefits/harms?
Better than alternatives?
Better identifiable subgroups?
Cost–Effectiveness Is there sufficient value?
Efficacy:
Effectiveness:
Net benefit optimal conditions
Net benefit average conditions
Penn
Penn
Assessment of Medical Interventions
Assessment of the most effective and efficient
care, defined in terms of patient outcome and cost.
• How much benefit and value?
• In which patients?
• Under which conditions?
Penn
“… it’s important to understand the methods
behind the research in order to know
whether or not the results are useful to you.
If the wrong methods are used or if the right
methods are misused, research results
won’t lead to better decisions but instead
could cause greater confusion.”
Sherine Gabriel, MD, MSc; Sharon-Lise Normand, PhD. PCORI Methodology
Committee, 2012
Penn
Randomized Clinical Trials:
The “Efficacy” Reference Standard
Random Rx allocation protects internal validity
• Internal validity
• Adjustment for confounders
• Data reliability
Random treatment allocation protects internal
validity, valid estimates of causal effects and
etiologic relationships for the sample at hand
Penn
Randomized Clinical Trial:
Clinical Scenario
• High validity required
• Significant selection bias / confounding
• Ethical issues not prevent randomization (e.g.,
clinical equipoise)
• Sufficient resources (subjects; funds)
• Clinically relevant outcomes measurable
• Time frame adequate RE: clinically relevant
outcomes; can await results; will still be relevant
Penn
Randomized Clinical Trials:
Representativeness and Data Limitations
Representativeness/Selection biases
• Eligibility criteria; Clinically–relevant subgroups
• Interventions compared
• Population studied (clinical/demographic
characteristics; practice settings)
Data Limitations
• Data quality (informative censoring; crossover;
protocol violations; missing data)
• Outcomes assessed (clinical relevance; range)
• Time horizon / Follow–up
Penn
• Resource use/ Cost
“The paradox of the clinical trial is
that it is the best way to assess
whether an intervention works,
but arguably the worst way to assess
who will benefit from it.”
Mant. Lancet. 1999;353:743–746.
Penn
“The benefit or harm of most treatments
in clinical trials can be misleading and
fail to reveal the potentially complex mixture
of substantial benefits for some,
little benefit for many, and
harm for few.”
– R Kravitz, Milbank Quarterly, 2004
Penn
Randomized Clinical Trials:
Capacity and Logistical Challenges
• Ethical issues often interfere with randomization
• Insufficient resources
– Too many questions
– Science changes too frequently
– Not enough subjects
– Not enough money
– Time horizons too long
Penn
What do I think about
comparative effectiveness,
RCT evidence based
clinical practice guidelines?
Penn
“I think it would be a good idea.”
– Mahatma Gandhi
[when asked what he thought
of Western civilization]
Penn
“Without major changes in how we
conceive, design, conduct, and analyze RCTs,
the nation risks spending large sums of money
inefficiently to answer the wrong questions—
or the right questions too late.”
Luce BR, Kramer JM, Goodman SN, Connor JT, Tunis S, Whicher D, Schwartz JS.
Rethinking randomized clinical trials for comparative effectiveness research: The need
for transformational change. Annals of Internal Medicine. 2009;151:206-209.
Penn
“Decisions about the use of therapeutic
interventions, whether for individuals or entire
healthcare systems, should be based on the
totality of available evidence. The notion that
evidence can be reliably or usefully placed in
‘hierarchies’ is illusory. Rather, decision makers
need to exercise judgement about whether (and
when) evidence gathered from experimental or
observational sources is fit for purpose.”
Rawlins M. De Testimonio – On the evidence for decisions
about the use of therapeutic interventions. Lancet.
2008;372:2152–2161.
Penn
Assessment of Medical Interventions:
Fit for purpose
• Start with how information will be used
• Identify gaps / required evidence generation
– Clinically relevant patients / subpopulations
– Clinically relevant, validated outcomes
• Sequence /integrate analytic approach within
and across studies
• Systematically, explicitly analyze incremental
tradeoffs/uncertainties across subgroups,
relevant outcomes and studies
• Bound summary estimates/CI; Identify
conditions/variables that drive decisions
Penn
Observational Studies
• Estimate causal interpretations from data
constructed for other purposes
• Random probability sample selection protects
external validity
• Outcomes not able to be adequately addressed
in a timely fashion by RCTs
– High need for representativeness that does
not perturb “routine care”
– Confident confounding can be adjusted
adequately statistically
– Outcomes adequately measured
Penn
Analysis of Medical Interventions:
Required Considerations
• Clinically relevant patients and subpopulations (baseline
risk; outcomes; preferences) and study settings
• Clinically relevant, validated outcomes
(mortality/morbidity; PROs; preferences, economic)
• Clinically relevant settings, behaviors, time horizons
• Explicit assessment incremental tradeoffs/uncertainties
for relevant populations/outcomes
• Improved methods RE: indirect comparisons;
confounding adjustment
• Transparent, flexible, clinically meaningful/relevant
assumptions and models
• Summary estimates; Bounded estimates/CI; Identification
of conditions/variables that drive decisions
Penn
Analysis of Medical Interventions:
Required Considerations
Broad range of empiric data and methods
• Experimental (RCT; pragmatic/practical)
• Observational (case–control, cohort, registry,
administrative claims, EMR, clinical networks)
• Synthesis (meta–analysis; systematic review)
• Integration (formal, structured, transparent,
expanded, validated, multi–factorial outcomes
models)
Penn
Challenges Developing the Evidence Base
• Serious errors in clinical practice may result
from overreliance on narrowly focused RCTs,
observational studies, data synthesis or
modeling
• Choice of study design and analytic approaches
involves tradeoffs among limitations inherent to
each and must be optimized for research
question
• Consistency across methods and cross–
validation confirmatory; disagreement requires
understanding
Penn
Formal Consideration of Patient
Preferences and Utilities
Potential Approaches/Opportunities
• Explicit quantification for outcomes about which
patients care
• Model to assess sensitivity of recommendations
to alternative preferences
Challenges:
• Whose utilities?
• Distribution of utilities
• Dissimilar trade-offs
• Misinformation/ “Popularity paradox”
Penn
Clinical Decision Making:
Threshold Theory
Test
Threshold
No Rx
0
Treatment
Threshold
TEST
Probability of Disease
Rx
1
Pauker SG, Kassirer JP. The threshold approach to clinical decision
making. N Engl J Med. 1980;3022:1109-1117.
Penn
Penn
Penn
Life is full of calculated risks…
Penn
Occupational Risks
Penn
Avoidable Risks
Penn
Unexpected Risks
Penn
Absolute vs. Relative Risk
“All policy decisions should be based on
absolute measures of risk;
relative risk is strictly for researchers only.”
– Geoffrey Rose
Professor Epidemiology
London School of Hygiene and Tropical Medicine
Penn
“… judgment cannot be excised from the process
of evidence synthesis and that the variation of
this judgment among experts generates
uncertainty just as real as the probabilistic
uncertainty of statistical calculations.”
Goodman SN. The mammography dilemma: A crisis for evidence-based medicine? Ann
Intern Med. 2002;137:363-365.
Penn
Evidence Based Medicine Is Not Value Free
• Harms and benefits involve comparison of
dissimilar outcomes, often from different data
sources with different populations, biases,
ascertainment methods and time horizons
• Subjective expertise – locus of control shifted
from physician to methodologist
Penn
"Not everything that
can be counted counts;
not everything that
counts can be counted.”
– Albert Einstein
Penn
The Pirate’s Code
Elizabeth: Wait! You have to take me to shore.
According to the Code of the Order of the
Brethren...
Barbossa: First, your return to shore was not
part of our negotiations nor our agreement so I
must do nothing.
And secondly, you must be a pirate for the
pirate's code to apply and you're not.
And thirdly, the code is more what you'd call
"guidelines" than actual rules.
Penn
“In the midst of every challenge
lies opportunity”
– Albert Einstein
Penn
Penn