Transcript Certificate session on Evidence

Evidence-Based Medicine and
Critical Appraisal
Ben Rehman
Director
London Medicines Information Service
Northwick Park Hospital
Introduction

Evidence-Based Medicine
• What is it?
• Why do we need it?
• How do we do it?


Critical appraisal—RCTs
Brief introduction to other study designs including
types of pharmacoeconomic studies
• How should we use it?



At patient level?
As part of broader decision making and resource
optimisation?
Example critical appraisal workshop
Evidence-Based Medicine—
What is it?




the conscientious, systematic, explicit, and
judicious use of current best evidence in
patient care
best research evidence replaces or
supports current practice
integration of evidence and patient or
population values
optimising health gain
• clinical evidence
• pharmacoeconomics
Evidence-Based Medicine—
Why do we need it?

Health professionals face problems
ensuring best possible patient care:
•
•
•
•


information overload
inadequate traditional sources
opinion based approach flawed
disparity between experience and up-to-date
knowledge
Healthcare systems have finite resources
But… it’s not a complete panacea!
Evidence-Based Medicine—
How do we do it?
1.
2.
3.
4.
5.
Form an answerable question
Obtain current best evidence
Critically appraise the evidence
Integrate appraised evidence with
clinical expertise and patient values
Evaluate our successes and failures
Evidence-Based Medicine—
How do we do it?
Evaluate successes
and failures—audit
Integrate evidence
and values—implement
changes
Formulate question
Track down
best evidence
Critically
review
quality
How do we do EBM?
1. Form an answerable question

Should contain:
• Intervention you are interested in
• Patient or population you are interested
in (clearly for drug interventions you
require the indication here too)
• Outcomes you are interested in

So for the example paper, our
original question might have been:
• Is adalimumab effective in patients with
Crohn’s disease previously intolerant to
or refractory to infliximab?
How do we do EBM?
2. Obtain current best evidence

Group discussion
• What are good sources of evidence?

What have you used in the past and found
useful?
• What makes them good?

What criteria do you apply?
• How can we find good evidence?

What portals and sources do we have
available?
How do we do EBM?
2. Obtain current best evidence

Discussion point: good sources of
evidence for EBM
How do we do EBM?
2. Obtain current best evidence


An appropriate search strategy is
important
We should be aware of the hierarchy
of evidence:
• Systematic reviews
• Meta-analyses
• Randomised-controlled trials
• Cohort studies
How do we do EBM?
2. Obtain current best evidence
The hierarchy of
evidence:
Its a guide—it’s not
necessarily this
simple! Consider:
• A large RCT vs. a
meta-analysis of small
RCTs?
• A meta-analysis of
observational studies?
Hierarchy becomes
complicated
How do we do EBM?
3. Critical appraisal

Consider the following concepts
• Internal validity

Does it prove what it set out to prove?
• External validity

Are the results true in the wider world?
• Bias

Is there systematic error?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs
Randomisation
Experimental
intervention
Population
Outcome
Sample
Control
intervention
Time
Outcome
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Are this study’s results valid?
• Did the trial address a clearly focused issue?

Consider the intervention and the outcomes
considered
• Is an RCT an appropriate method to answer
this question?
• Was the assignment of patients to treatment
randomised?




Why is it important?
Acceptable vs dubious methods?
Is the process well described?
Was the process concealed?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Are this study’s results valid?
• Were the groups similar at the start of the
trial?


Could any differences have affected outcome?
Was any method used to balance randomisation
(stratification)?
• Was follow-up sufficient?


Length
Completeness
• Was there sufficient power?

Were there enough participants to minimise the play
of chance?
• Was an intention-to-treat analysis performed?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Are the results of this individual study
important for us?
• Define the population studied

Inclusion and exclusion criteria
• Endpoints

Measurement of outcome
• Clinical relevance
• Surrogate and composite markers
• Validity

Primary vs. secondary
• Balance of beneficial outcomes against sideeffects
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Are the results of this individual study
important for us?
• What is the magnitude of the treatment effect?

How is it described?
• Proportions of people, a measurement (mean or
median differences), a survival curve?

How is it expressed?
• With proportions see terms like relative risk reduction,
absolute risk reduction, number needed to treat
• Is what is being expressed clear?
• Is it clinically significant?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Definitions for measures of
association and effectiveness
• Control event rate

the rate at which an outcome occurs in the control
population
• Experimental event rate

the rate at which an outcome occurs in the
experimental population
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Definitions for measures of association
and effectiveness
• Absolute risk (AR)


Probability an individual will experience a specified
outcome during a specified period
Range of 0 to 1, or expressed as a percentage
• Relative risk (RR)

How many times more likely (RR > 1) or less likely (RR <
1) an event is to happen in one group compared with
another
• Absolute risk reduction (ARR)

Absolute difference in risk between experimental and
control groups
• Relative risk reduction (RRR)

Proportional reduction in risk between experimental and
control participants in a trial
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Definitions for measures of
association and effectiveness
• Number needed to treat (NNT)



A measure of treatment effectiveness
Measures the people who need to be treated with an
intervention over a period of time to prevent an
additional adverse outcome or achieve an additional
beneficial outcome
Reciprocal of ARR
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Definitions for measures of association
and effectiveness
• Odds ratio (OR)



Measure of treatment effectiveness—likelihood of event
happening in experimental group vs. control group
Effects being measured may be adverse (e.g. death or
disability) or desirable (e.g. survival)
If events rare OR analogous to relative risk (RR); as
event rates increase the OR and RR diverge
• When interpreting remember:




Closer OR is to 1  smaller the difference in effect
between experimental intervention and control
intervention
OR > 1 ≡ effects of treatment more than control
OR < 1 ≡ effects of treatment less than control
OR 95% CI includes 1 ≡ effect cannot be deemed
statistically significant
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Absolute vs. relative risk reduction as
measures—an example
• Pros and cons of each measure



Clinical significance?
Proportional difference?
Ease of interpretation?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs
An example—4S study (Scandinavian
Simvastatin Survival Study)1

• Purpose
Evaluate effect of cholesterol lowering with simvastatin
on mortality and morbidity in patients with coronary
heart disease (CHD)
• Participants
Patients with stable angina or history of myocardial
infarction more than 6 months previously
Serum cholesterol > 6.2mmol/l
Excluded patients with arrhythmias and heart failure
Run in of 8 weeks of dietary therapy
If cholesterol still raised (5.5-8.0 mmol/L) randomised
to receive simvastatin (20mg esculating to 40mg) or
placebo
1.
Lancet. 1994 Nov 19;344(8934):1383-9
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

4S study
• Randomisation and allocation
4444 patients randomised to double-blind
treatment with simvastatin or placebo
2223 patients placebo
2221 were simvastatin
Mean follow-up was 5.4 years
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

4S study
• Results
• All cause mortality:
256 patients (12%) in placebo group dead vs 182 (8%)
in simvastatin group
• Coronary deaths:
189 coronary deaths in the placebo group vs. 111 in the
simvastatin group
• Morbidity (coronary event):
622 patients (28%) in placebo group vs. 431 (19%) in
the simvastatin group
• So, on the face of it, the results look extremely
promising. How do the measures of association we
discussed earlier play out?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

4S study results—all cause mortality
OUTCOME
Dead
Alive
Control group (placebo) n = 2223
256
1967
Experimental group (simvastatin) n = 2221
182
2039
Control event rate
= 256 =
2223
0.115 (11.5%)
Control odds of event
= 256 =
1967
0.130
Experimental event rate
= 182 =
2221
0.082 (8.2%)
Experimental odds of event
= 182 =
2039
0.089
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

4S study results—all cause mortality
Relative risk reduction
RRR
CER – EER
CER
= 0.115 – 0.082 = 0.29 (29%)
0.115
Absolute risk reduction
ARR
CER – EER
= 0.115 – 0.082 = 0.033 (3.3%)
Number needed to treat
NNT
_1_
ARR
=
_1_
0.033
=
30.1
Relative risk
RR
EER
CER
=
0.082
0.115
=
0.71
Odds ratio =
OR
odds of event in experimental group
odds of event in control group
= 0.089 =
0.130
0.69
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

4S study results—all cause mortality
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

NNT examples—when should we adopt therapy?
Intervention
Streptokinase + aspririn v.
placebo (ISIS 2)
tPA v. streptokinase
(GUSTO trial)
Simvastatin v. placebo in IHD
(4S study)
Treating hypertension in the
over-60s
Aspirin v. placebo in healthy
adults
Outcome
NNT
prevent 1 death
at 5 weeks
save 1 life with
tPA usage
prevent 1
event in 5y
prevent 1 event
in 5y
prevent MI or
death in 1 year
20
100
15
18
500
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Discussion point: numbers needed to
treat
• When do you think therapy should be
adopted?
• Clearly there is no one answer to this
but what might be the considerations?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Statistical approaches to uncertainty
• Why do they exist?


Cannot include all individuals in a population
in trial
Need to quantify uncertainty
• p values and confidence intervals are
the measures used, but what are they
and what are the differences?
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Statistical approaches to uncertainty
• p values





assess the significance of the difference between
a sample estimate and a hypothesised value
tell us the probability that an observed effect
occurred by chance if in truth there is no effect
But… doesn’t quantify the size of an effect, nor
the direction
p<0.05 → commonly reject null hypothesis
ideally p<<<0.05 and trial reports the actual
value
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Statistical approaches to uncertainty
• Confidence intervals



Range of values around the sample estimate
(i.e. the value found in the study)
The range has specified probability (usually
95%) so CI acts as hypothesis test for the
range
Should be seen with most measures of
association
How do we do EBM?
3. Critical appraisal—placebo
controlled RCTs

Statistical approaches to uncertainty—
please remember!
• statistical significance ≠ clinical significance…
what is actually being measured?
• sample size is still important

larger sample = less uncertainty = narrower CI
and smaller observed effect considered significant
• there can still be error


rejecting null hypothesis when it’s true (type 1)
Not rejecting the null hypothesis when it’s false
(type 2)
How do we do EBM?
3. Critical appraisal—RCTs of
therapeutic equivalence



Increasingly used by pharma.
Designed to show a new treatment is
not inferior to standard treatment by
a predefined clinically acceptable
endpoint
Rely on assumptions that can be
hard to validate
How do we do EBM?
3. Critical appraisal—RCTs of
therapeutic equivalence

Trials based on the concept that:
• new treatment is non-inferior
• but would exhibit therapeutic efficacy if
a placebo controlled RCT could/was
performed
• new treatment offers ancillary benefits

Note: non-inferior so efficacy
determined by effect against placebo
not comparator
How do we do EBM?
3. Critical appraisal—RCTs of
therapeutic equivalence

Estimation of non-inferiority margin
• Use statistical and clinical reasoning to
determine what is non-inferior
• If there are a variety of placebo
controlled RCTs this will be similar to a
meta-analysis

Non-inferiority margin determined as
a fraction (f) and specifies an
acceptable magnitude for treatment
effect that must be preserved
How do we do EBM?
3. Critical appraisal—RCTs of
therapeutic equivalence

What to look for
• These trials should be designed in a manner
consistent with the historical placebo-controlled
trials
• The active comparator should be well established
with predictable quantifiable, and consistent effects
• What constitutes non-inferiority should be
determined prior to initiation of trial
• Protocol deviations and poor adherence may have a
larger impact on quality than in conventional trials
• Analysis should be by intention-to-treat AND ontreatment, and should also report absolute AND
relative effects
How do we do EBM?
3. Critical appraisal—RCTs of
therapeutic equivalence

So when appraising these trials ask
yourself:
• Was the active control previously shown to be effective?
• Were study patients and outcome variables similar to
those in the original trials that established the efficacy of
the active control?
• Were both regimens applied in optimal fashion?
• Was the appropriate null hypothesis tested?
• Was the equivalence margin specified before the study?
• Was the trial large enough?
• Was the analysis intention-to-treat AND on-treatment,
and should also report absolute AND relative effects?
• PLUS usual assessment of size and precision of
treatment effect!
How do we do EBM?
3. Critical appraisal—other
study designs


Many other types of study in the
hierarchy e.g. systematic reviews,
case-control studies, cohort studies
Details of appraisal not included
today but worth thinking about most
in terms of:
• Internal validity
• Bias
• External validity
How do we do EBM?
3. Critical appraisal—other
study designs

Cohort study
Exposed
Population
Sample
Outcome
Time
Not exposed
Outcome
How do we do EBM?
3. Critical appraisal—other
study designs

Case control study
Study
Exposed
Cases
Not exposed
Population
Exposed
Controls
Not exposed
Time
How do we do EBM?
3. Economic evidence




Important in decision making
Considers costs and consequences of
alternate courses of action
Scope depends on level of decision
making they inform
Costs always measured the same
way (although scope may vary),
measurement of consequence varies
according to study type
How do we do EBM?
3. Critical appraisal—what is
economic evidence?

Cost minimisation analyses
• analyse difference in costs where there
is no difference in outcome
• narrow focus

Cost-effectiveness analyses
• outcome expressed in natural units (e.g.
validated single marker)
• again fairly narrow scope—generally 2
interventions for single indication
How do we do EBM?
3. Critical appraisal—what is
economic evidence?

Cost-utility analyses
• Costs and consequences but
consequence measured as utility
• Utility = a value of health state rather
than a natural marker (usually QALY)
• QALY is a measure of a person’s length
of life weighted by a valuation of their
health-related quality-of-life (HRQL)
over that period
• Can compare incongruent interventions
How do we do EBM?
3. Critical appraisal—what is
economic evidence?

Cost utility analysis
• Used by NICE




Consider outcome in terms valuable to patients i.e.
life expectancy and quality
Uses a standard measure (utility)
Comprehensive consideration of costs
NICE state that:
Technologies can be considered to be cost
effective if their health benefits are greater
than the opportunity costs measured in terms
of the health benefits associated with
programmes that may be displaced to fund the
new technology. In other words, the general
consequences for the wider group of patients in
the NHS are considered alongside the effects
for those patients who may directly benefit
from the technology of interest.
How do we do EBM?
3. Critical appraisal—what is
economic evidence?

Cost-benefit analyses
• Broadest possible scope!
• Allocation of resources between
difference sectors of economy
• Maximising social welfare
• Rely on assigning cost to healthcare
intervention—incredible complicated in
practice, particular in NHS where
individuals don’t pay at point of delivery
How do we do EBM?
3. Critical appraisal—further
reading

A lot is published! But texts I’ve found
useful include:
• Straus SE, Richardson WS, Paul Glasziou, Haynes RB.
Evidence-based Medicine: How to Practice and Teach
EBM, Third Edition. Churchill Livingstone: Edinburgh,
2005
• Guyatt GH, et al. Users’ Guides to the Medical
Literature: II. How to use an article about therapy or
prevention. A. Are the results of the study valid? JAMA
1993; 270: 2598-2601
• Guyatt GH, et al. Users’ Guides to the Medical
Literature: II. How to use an article about therapy or
prevention. B. What are the results and will they help
me in caring for my patients? JAMA 1994; 271: 59-63
• Evidence Based Medicine (EBM) journal notebook
series—available online
• Clinical evidence online. Excellent terminology glossary
and explanations
How do we use EBM?
4. Applying results of research to
our individual patient or population





Is our patient/population so different from
those in the study that its results cannot
apply?
Is the treatment feasible for us?
What are the potential benefits and harms
from therapy?
What are our patients’ values and
preferences for both the outcome and
side-effects?
Is there any economic analysis supporting
therapy?
Example critical appraisal
workshop

The scenario
• You are asked to review the evidence
and to contribute towards an individual
funding request for submission to a PCT.
• The patient has refractory Crohn’s
disease specified as being previously
intolerant to azathioprine,
methotrexate, salazopyrine, and
infliximab
Example critical appraisal
workshop

The background
• Adlimumab is licensed for severe active
Crohn’s unresponsive to corticosteroids
and conventional immunosupression
• Use in relation to infliximab is not well
defined. The SMC reviewed
adalimumab for Crohn’s in 2007 but not
consider place in therapy wrt infliximab
• A NICE TAG is pending
Example critical appraisal
workshop

Your role
• You have identified the GAIN study
• You now need to appraise it and to
reach a conclusion as its validity with
reference to the circumstance described
• You can use the CASP tool to help
Example critical appraisal of RCT
workshop

Working through each point from the CASP tool:
1. Did the study ask a clearly-focused question?
2. Was this a randomised controlled trial (RCT)
and was it appropriately so?
3. Were participants appropriately allocated to
intervention and control groups?
4. Were participants, staff and study personnel
‘blind’ to participants’ study group?
5. Were all of the participants who entered the
trial accounted for at its conclusion?
Example critical appraisal of
RCT workshop
6. Were the participants in all groups followed
up and data collected in the same way?
7. Did the study have enough participants to
minimise the play of chance?
8. How are the results presented and what is
the main result?
9. How precise are these results?
10. Were all important outcomes considered so
the results can be applied?
Example critical appraisal of
RCT workshop
Overall, what are the
main messages with
reference to this
patient?

Good points

Potential
drawbacks