Transcript Critical appraisal of clinical research evidence

Critical appraisal of clinical research
evidence
Chris Lewis – May 2008
How to read a “paper”
Objectives:
To enable VTS members to have a good working
knowledge of:
• Processes of EBM
• Conduct of a RCT
• Sources of bias in a RCT
• Risk assessment terminology (RR ARR NNT)
Evidence Based Medicine
Incorporating the best available research
evidence into clinical decision making
Processes of Evidence Based Medicine
• Asking answerable questions (PICO)
• Accessing the best information
• Appraising the information for validity and
relevance
• Applying the information to patient care
Asking an answerable question
•
•
•
•
Population
Intervention
Comparator
Outcome(s)
Types of “paper” research evidence
• Primary studies
–
–
–
–
Case studies
Experiments
Surveys
Clinical Trials
• Secondary studies
– Non-systematic reviews
– Systematic reviews
• Meta-analyses
• Guidelines
• Decision analyses
• Economic analyses
Types of evidence
Advantages
Summarises all relevant
research about all possible
interventions for a clinical
problem. Explores benefits
and harms.
Disadvantages
May become out-of-date
quickly.
Expert opinion often fills
gaps in evidence.
Systematic Review
Summarises all research
about an intervention.
Usually only one of several
possible interventions is
considered. May not
explore benfits vs harms.
Primary Study
Very specific information
Not comprehensive
Evidence-based
Guideline
Topics of Primary Study and Types of Study Design
Phenomena
Observation / qualitative studies
Aetiology
Cohort studies (or Case-control studies)
Diagnosis and screening
Cross-sectional analytical studies
Prognosis
Cohort studies
Intervention
Randomised Controlled Trials
How do you read a clinical research paper?
How to read a (clinical research) paper
• Scan abstract for a few seconds
–
–
–
–
Are the authors conclusions of interest?
Briefly assess study design
Briefly assess statistical precision of results
Formulate a brief summary
• Critically appraise methods & results sections for
validity
• Critically appraise results section (especially the
tables and figures) for relevance
• Draw your own conclusions about clinical
applicability
What conclusions have you drawn
concerning clinical application of the
Heart Protection Study?
How to read a (clinical research) paper
• Scan abstract for a few seconds
–
–
–
–
Are the authors conclusions of interest?
Briefly assess study design
Briefly assess statistical precision of results
Formulate a brief summary
• Critically appraise methods & results sections for
validity
• Critically appraise results section (especially the
tables and figures) for relevance
• Draw your own conclusions about clinical
applicability
What is the essential information you want to know about any clinical
research evidence?
WHAT INFORMATION WOULD YOU
INCLUDE IN A BRIEF SUMMARY OF A
CLINICAL RESEARCH PAPER?
Information to include in summary
•
•
•
•
•
•
•
•
•
Type of study
Size
Study Population
Intervention
Comparator
Duration
Outcome(s)
Main findings (with relevant statistics)
Conclusion(s)
Produce a brief summary for the
Heart Protection Study
(6 to 8 short sentences)
Note which parts of the paper you have to read
to produce your summary.
Heart Protection Study
(Lancet.v360.pp7-22.6/7/2002)
• Randomised placebo-controlled trial
• 20536 UK adults, aged 40 to 80, with CHD,
other occlusive arterial disease or diabetes.
• Effect of Simvastatin 40mg vs placebo on
mortality, fatal and non-fatal vascular events.
• 5 years follow-up
• All cause mortality reduced in the simvastatin group
1328/10269 (12.9%) vs 1507/10267 (14.7%); p=0.0003;
RR 0.87 (0.81-0.94); NNT=55
• First vascular event rate reduced in the simvastatin
group
2033/10269 (19.8%) vs 2585/10267 (25.2%); p<0.0001;
RR 0.76 (0.72-0.81); NNT=18.
• No significant harms identified
• All people similar to the study population should be
treated with Simvastatin 40mg
How much of the paper have we actually
read to get to this summary?
How to read a (clinical research) paper
• Scan abstract for a few seconds
–
–
–
–
Are the authors conclusions of interest?
Briefly assess study design
Briefly assess statistical precision of results
Formulate a brief summary
• Critically appraise methods & results sections for
validity
• Critically appraise results section (especially the
tables and figures) for relevance
• Draw your own conclusions about clinical
applicability
Flow Diagram for a RCT / cohort study
1 Selection and sampling
2 Allocation (with or without Randomisation)
3 Follow-up
4 Outcomes
5 Analysis
Validity - External Validity
To whom do the results of this trial apply?
Can the results be reasonably applied to a
definable group of patients in a particular
clinical setting in routine practice?
Are the results generalisable beyond the trial
setting?
Appraisal of External Validity
• Where were the participants recruited from
(primary care / referral centre)?
• Do the inclusion and exclusion criteria make
sense?
• What proportion of the screened population
was recruited?
Where were the participants recruited from?
Where were the participants recruited from?
Methods: Recruitment p8
• 69 UK hospitals
Inclusion Criteria:
Inclusion Criteria:
• Methods: Eligibility p8
• Men and women aged 40 to 80 +
• Blood total cholesterol >= 3.5mmol/L +
• Past medical history of any one or more of
CHD, CVA, TIA, PVD, DM
OR
• Men, 65 to 80, treated for hypertension
Exclusion criteria:
Exclusion criteria:
1.
Anyone already on a statin or
Dr considered statin to be
clearly indicated.
2. Contraindications
– Chronic liver disease
– ALT >67 IU/L (1.5 x ULN)
– Child-bearing potential
3. Conditions requiring a
dose reduction
– Severe renal disease
– Creatinine >200 mmol/L
4. Interactions
– Treatment with ciclosporin,
fibrates, niacin
5. Conditions similar to
known unwanted effects
– Inflammatory muscle disease
– CK >750 IU/L (3 x ULN)
6. Patient unlikely to survive
5 years follow-up
– Severe heart failure
– Another life threatening
condition
7. Conditions limiting
compliance
– Severely disabling stroke
– Dementia
What proportion of the screened
population was recruited?
What proportion of the screened population
was recruited?
Results: patient enrolment & Fig.1 p10
• 49% (31458/63603) of screened population excluded or refused
We are not given a break-down of the reasons
• 36% (11609/32145) of population accepted for run-in were not
subsequently randomised.
– 26% chose not to enter or “did not seem likely to be compliant for 5
years”
– 5% considered to have clear indication for statin
– 3% raised ALT, CK or Creatinine at pre-treatment screen
– 2% attributed various problems to run-in treatment
– 1% cholesterol <3.5mmol/L
• Only 32% (20536/63603) of screened population were randomised.
Validity - Internal Validity
The extent to which the observed difference
in outcomes between the two comparison
groups can be attributed to the intervention
rather than other factors.
What are the possible causes of an
“effect” in a RCT?
What are the possible causes of an
“effect” in a RCT?
•
•
•
•
Bias
Placebo
Chance
Real effect
Bias
• Allocation (Selection) Bias – Failure of randomisation
Systematic differences in comparison groups
• Performance Bias
Systematic differences in interventions received by the two
groups
• Attrition Bias
Systematic differences in withdrawals from the trial
• Detection (Measurement) Bias – Failure of blinding
Systematic differences in outcome assessment
Internal Validity - Sources of Bias in a RCT
2 Allocation Bias (Failure of Randomisation)
3 Follow-up – Performance Bias and Attrition Bias
4 Outcomes – Detection Bias (Failure of Blinding)
CONSORT definition:
Selection bias—a systematic error in creating
intervention groups, causing them to differ
with respect to prognosis. The groups differ in
measured or unmeasured baseline
characteristics because of the way in which
participants were selected for the study or
assigned to their study groups.
Confounding—a situation in which the estimated
intervention effect is biased because of some
difference between the comparison groups apart
from the planned interventions - such as baseline
characteristics, prognostic factors, or concomitant
interventions.
For a factor to be a confounder, it must differ
between the comparison groups and predict the
outcome of interest.
Comparison of Cohort and RCT
Cohort
• Population diverse
• Allocation by clinical decision
• Outcomes can be defined
retrospectively
• Outcomes may be rare
• Follow-up may be
retrospective and may be
long-term
• Analysis complex multivariate
RCT
• Population highly selected
• Allocation by chance
• Outcomes defined
prospectively
• Outcomes must be common
• Follow-up pre-determined
and usually short-term
• Analysis relatively simple
Possible types of comparisons in cohort study
• General population
– Intervention v alternative intervention
– Intervention v no intervention
• Restricted population
– Intervention v alternative intervention
– Intervention v no intervention
Do patients who receive atypical antipsychotic drugs have an
increased risk of hip fracture?
All older people
Older people with dementia
Atypical
No
Atypical
No
antipsychotic Intervention antipsychotic intervention
(n=34 960) (n=1 251 435) (n=21 427)
(n=58 754)
Mean (SD) age 80.46 (7.63)
No (%) with
21 427 (61.3)
dementia
74.50 (6.58)
58 754 (4.7)
81.69 (7.11)
21 427 (100)
80.95 (7.64)
58 754 (100)
Effect on age distribution and sample size of restricting comparison of
atypical antipsychotic with no intervention to individuals with dementia
Consider the difference between the three sets of figures here:
Atypical antipsychotic
(n=21 427)
No intervention
(n=58 754)
Mean (SD) age
81.69 (7.11)
80.95 (7.64)
Mean (SD) age
81.69 (1.11)
80.95 (7.64)
Mean (SD) age
81.69 (7.11)
80.95 (1.64)
Selection / Allocation Bias
Assessed by looking at the
Table of Baseline Characteristics
Womens Health Initiative
JAMA v288, pp321-333, 17th July 2002
• A randomised placebo-controlled trial.
• 16608 American women, aged 50-79, with intact
uterus.
• Effect of conjugated equine oestrogens 0.625mg
od + medroxyprogesterone acetate 2.5mg od on
incidence of CHD and Breast Cancer
• 8.5 years follow-up planned, but stopped after
5.2 years.
• CHD rate increased in oest+prog group
164/8506 (1.93%) vs 122/8102 (1.51%); RR 1.29 (1.02-1.63);
ARI 0.42%; NNH 238
• Breast Ca rate increased in oest+prog group
166/8506 (1.95%) vs 124/8102 (1.53%); RR 1.26 (1.00 – 1.59);
ARI 0.42%; NNH 238
• Treatment group also had increased rate of venous thromboembolism and reduced rates of fractures and colo-rectal carcinoma.
• Overall long-term harms exceeded benefits
751/8506 (8.83%) vs 623/8102 (7.69%); RR 1.15 (1.03-1.28);
ARI 1.14%; NNH 88
• When prescribing combined HRT in the over-50’s short-term
benefits should be balanced by consideration of long-term harms.
Table of Baseline Characteristics (WHI)
• Are all important characteristics listed?
• Are any of the differences between the treatment
and placebo groups statistically significant?
• Are there any differences in the two groups that may
bias the results?
• What age range includes 95% of the Placebo group?
• Assuming HRT has no effect – which group would
you expect to have more heart attacks?
• Does this introduce a bias?
• If so, in which direction does it operate?
Performance Bias
• Contamination:
Provision of the intervention to the control group
• Compliance:
Poor compliance with the allocated intervention
• Co-interventions
Provision of unintended additional interventions
to either group
Attrition Bias
• Count (drop-out)
Loss to follow-up rate
should not exceed outcome event rate and
should be equal in all groups.
Detection (Measurement) Bias
Best: Double-blind
Both patient and investigator unaware of
treatment allocation
Less important if outcome is objective (e.g.
death)
Critical if outcome is subjective
Impossible for some comparisons eg medical vs
surgical intervention
What are the possible causes of an
“effect” in a RCT?
•
•
•
•
Bias
Placebo
Chance
Real effect
Placebo Effect
You can only know the size of a placebo effect
if a placebo has been used!
Appraisal of Internal Validity
• Was assignment of patients to treatments randomised?
• Were groups similar at start of trial?
• Were groups treated similarly, apart from the
experimental treatment?
• Were all participants accounted for in the conclusions?
• Were all participants analysed in the groups to which
they were randomised (Intention To Treat analysis)?
• Were participants and clinicians kept “blind” to
treatment received?
Randomisation?
(with allocation concealment)
Best: Centralised computer randomisation
Should be independent of investigators
Where: Methods
HPS
Methods: Recruitment p8
“The central telephone randomisation system…”
Similar Groups?
Table of baseline characteristics
(Note p-values)
Where: Results
HPS:
Methods: Recruitment p8
“The central telephone randomisation system used a minimisation algorithm to balance the
treatment groups with respect to eligibility criteria and other major prognostic factors”
Results: Patient Enrollment p10
“…good balance between the groups for the main pre-randomisation prognostic features…”
But – there is no table of baseline characteristics!
We are left to make our assessment from the numbers allocated to each sub-group in Fig.8 p16
Treated equally?
Where:
Methods - for intended schedule
Results - for actual treatment
HPS:
Methods: Recruitment p8
“..randomly allocated to receive 40mg simvastatin
daily or matching placebo tablets in specially
prepared calendar packs..”
HPS:
Results: Compliance p11
Placebo-allocated group more likely to be
prescribed a non-study statin
(32% vs 5% at end of study).
Averaged over the 5 years of study 85% of
simvastatin group and 17% of placebo group
received a statin.
Non-equal treatment – Classic example
1948 – Trial of Vitamin E in pre-term infants
Vitamin E prevented retrolental fibroplasia
(by removal from 100% Oxygen to give the
frequent doses of Vit E)
Loss to follow-up
Rough guide: 5% - OK
>20% - validity doubtful
AND
Must not exceed outcome event rate
Where: Results
HPS – Loss to Follow-up
Results: Fig.1 p10
Mortality
Morbidity
Total
Outcome ER
Simvastatin
0.03%
0.33%
0.36%
19.8%
Placebo
0.04%
0.25%
0.29%
25.2%
Intention-to-treat analysis
Maintains the randomisation
Where: Results
HPS:
Summary: Methods p7
“Analyses…compare all simvastatin-allocated
versus all placebo-allocated participants. These
‘intention-to-treat’ comparisons…”
Results: Compliance p11
Blinding?
Best: Double-blind
Where: Methods
HPS:
Does not mention “blinding”
But we are given some suggestion the randomisation process kept the
allocation concealed from both patients and investigators.
Methods: Recruitment p9
“…matching placebo tablets in specially prepared calendar packs.”
Methods: Follow-up p9
“….coordinating centre clinical staff…were kept unaware of the study
treatment allocation.”
How to read a (clinical research) paper
• Scan abstract for a few seconds
–
–
–
–
Are the authors conclusions of interest?
Briefly assess study design
Briefly assess statistical precision of results
Formulate a brief summary
• Critically appraise methods & results sections for
validity
• Critically appraise results section (especially the
tables and figures) for relevance
• Draw your own conclusions about clinical
applicability
Appraisal of Relevance / Impact
•
•
•
•
Were all the outcomes studied important?
Were all the important outcomes studied?
Was sub-group analysis pre-planned?
Could the treatment effect have arisen by
chance?
• How large was the treatment effect?
HPS: Outcomes
• Primary Outcomes
– Mortality
– Non-fatal vascular events
• MI, CVA, Revascularisation
• Secondary Outcomes
– Cancer
– Other major morbidity
Where: Summary
Was sub-group analysis pre-planned?
Where: Methods, Statistical Analysis
“The data analysis plan was
prespecified…before any analysis of the
effects of treatment were available…”
Could the treatment effect have arisen by chance?
p-values
Statistical test of the (“null”) hypothesis that the
intervention had no effect
If p<0.05 result is statistically significant
i.e. the effect would occur by chance less than 5% of the time
The smaller the p-value the less likely is the effect to
occur by chance
Confidence Intervals (95%)
Range of values that has a 95% chance of including the
true value
How large was the treatment effect?
Death
Event
RR
0.87
0.76
RRR
0.13
0.24
ARR
1.8%
5.4%
NNT
55
18
Expressions of Risk
In the study population, treatment with Simvastatin 40mg
for 5 years, compared with placebo, resulted in:
• A 13% reduction in risk of death
• A 24% reduction in risk of a major vascular event
• A 1.8% reduction in deaths
• A 5.4% reduction in major vascular events
• We need to treat 55 people to defer one death
• We need to treat 18 people to prevent / defer a major
vascular event.
Expressions of Risk
•
•
•
•
Relative Risk (RR) = EER / CER
Relative Risk Reduction (RRR) = 1 – RR
Absolute Risk Reduction (ARR) = CER – EER
Numbers Needed to Treat (NNT) = 1/ARR
EER = Experimental Event Rate
CER = Control Event Rate
Relative Risk
RR & RRR may remain constant despite huge
differences in absolute event rates
They are useful to determine whether a biological
effect exists ….
BUT
They do not discriminate between huge
treatment effects and trivial ones.
CER
EER
RRR
0.16
0.10
37.5%
0.016
0.010
37.5%
0.0016 0.0010 37.5%
Absolute Risk Reduction
ARR reflects baseline risk and does
discriminate between huge and trivial
treatment effects.
CER
EER
RRR
ARR
0.16
0.10
37.5% 6%
0.016
0.010
37.5% 0.6%
0.0016 0.0010 37.5% 0.06%
Numbers Needed to Treat
The number of people you need to treat
for one of them to have the desired outcome
over a specified period of time.
A good measure of clinical relevance.
Allows calculation of cost per desired outcome.
CER
EER
RRR
ARR
NNT
0.16
0.10
37.5%
6%
16.7
0.016
0.010
37.5%
0.6%
167
0.0016
0.0010
37.5%
0.06%
1667
NNT for various CER’s and RRR’s
CER
0.9
0.3
0.1
0.01
0.001
50%
2
7
20
200
2000
40%
3
8
25
250
2500
RRR
30%
4
11
33
333
3333
20%
6
17
50
500
5000
10%
11
33
100
1000
10000
Note that a small RRR for a condition with a high CER is more clinically
important than a large RRR for a condition with a low CER
Cost: HPS
Can be calculated from NNT
• It costs (£25 x 12 x 5 x 55) £82500 to defer one
death.
• It costs (£25 x 12 x 5 x 18) £27000 to prevent /
defer one major vascular event.
Women’s Health Initiative (WHI)
Estrogen +
Progestin
Placebo
Women (n)
8506
8102
Fractures ~5 yrs
650
788
Rate (annualised)
0.0147
0.0191
Can you calculate RR, RRR, ARR and NNT?
Women’s Health Initiative (WHI)
Estrogen +
Progestin
Placebo
Women (n)
8506
8102
Fractures ~5 yrs
650
788
Rate (annualised)
0.0147
0.0191
RR = EER/CER = 0.0147/0.0191 = 0.77
RRR = 1-RR = 1 – 0.77 = 0.23
ARR = CER–EER = 0.0191 – 0.0147 = 0.0044
NNT = 1/ARR = 1/0.0044 = 227
Women’s Health Initiative (WHI)
HRT for one year:
• Reduces the risk of fracture by 23%
• Reduces the number of fractures by 0.44%
• We need to treat 227 post-menopausal
women for one year to prevent one fracture
At ~£20/month (20x12x227) =
£54480 per fracture prevented
Number Needed to Harm
Women (n)
Breast Cancer ~5 yrs
Rate (annualised)
Estrogen +
Progestin
Placebo
8506
166
0.0038
8102
124
0.0030
Can you calculate RR, RRI, ARI, NNH?
Number Needed to Harm
Women (n)
Breast Cancer ~5 yrs
Rate (annualised)
Estrogen +
Progestin
Placebo
8506
166
0.0038
8102
124
0.0030
RR = EER/CER = 0.0038/0.0030 = 1.27
RRI = RR-1 = 1.27-1 = 0.27
ARI = EER-CER = 0.0038-0.0030 = 0.0008
NNH = 1/ARI = 1250
Women’s Health Initiative (WHI)
HRT for one year:
• Increases the risk of breast cancer by 27%
• Increases the number of breast cancers by
0.08%
• We need to treat 1250 post-menopausal
women for one year to give one of them
breast cancer.
How to read a (clinical research) paper
• Scan abstract for a few seconds
–
–
–
–
Are the authors conclusions of interest?
Briefly assess study design
Briefly assess statistical precision of results
Formulate a brief summary
• Critically appraise methods & results sections for
validity
• Critically appraise results section (especially the
tables and figures) for relevance
• Draw your own conclusions about clinical
applicability
Appraising Applicability
• Is my patient similar to the study population?
• Is the treatment feasible in my clinical setting?
• Will potential benefits of treatment outweigh
potential harms of treatment for my patient?
www.nntonline.net
Outcomes with CER 25% and NNT=20
Worse
with Rx
CURE. NEJM 2001 v345 p394
Discuss the evidence for and against the following
conclusions regarding clinical applicability in relation to
the Heart Protection Study:
Group 1
• There is no need to check LFT’s prior to commencing
statin therapy.
• There is no need to check LFT’s during statin therapy.
• There is no need to check CK prior to commencing
statin therapy.
• There is no need to check CK during statin therapy.
Group 2
• All people with diabetes should be treated
with a statin.
• All men >65 who are treated for hypertension
should be treated with a statin.
Group 3
• Amongst people similar to the study population
there is no need to test blood cholesterol levels
prior to commencing a statin.
• Once a decision has been taken to start statin
treatment there is no need to monitor blood
cholesterol levels.
Group 4
• Statins should be stopped at age 80.
• Women benefit from statin treatment to the
same extent as men.
• Once started statin treatment should be
continued indefinitely.
Group 1
• There is no need to check LFT’s prior to commencing
statin therapy.
• There is no need to check LFT’s during statin therapy.
• There is no need to check CK prior to commencing
statin therapy.
• There is no need to check CK during statin therapy.
• People with raised ALT and CK were excluded
from the study.
• We are not told how many screened people
were excluded for this reason, but 3% of prerandomisation run-in group were excluded for
raised ALT, CK or Creatinine (Results para1
p10)
ALT 2-4x
ALT >4x
CK 4-10x
CK >10x
Myo
Rhabdo
Persistant
ALT >4x
Persistant
CK >4x
Simva
1.35%
0.42%
0.19%
0.11%
0.05%
0.05%
Placebo
1.28%
0.31%
0.13%
0.06%
0.01%
0.03%
ARI
0.07%
0.11%
0.06%
0.05%
0.04%
0.02%
NNH
1428
909
1667
2000
2500
5000
p
0.09%
0.04%
0.05%
2000
0.3
0.07%
0.01%
0.06%
1667
0.07
0.2
• There is good evidence that it is not necessary to
monitor LFT’s or CK in follow-up of Simvastatin 40mg
treatment in the absence of relevant symptoms.
• The low (not statistically significant) attributable risk of
persistently raised ALT or CK due to Simvastatin 40mg
makes it improbable that pre-treatment testing will be
of value to the patient.
• This study does not provide evidence about
generalisability to other statins (or other doses of
simvastatin).
Group 2
• All people with diabetes should be treated
with a statin.
• All men >65 who are treated for hypertension
should also be treated with a statin.
Event
Simva
rate
Diabetes 20.2%
Placebo ARR
NNT
25.1%
20
4.9%
There is good evidence that people with diabetes
aged 40 to 80, with total cholesterol >=3.5 mmol/L
should be offered Simvastatin 40mg.
• Only 1% of the trial population was male >65
on treatment for hypertension and had no
history of vascular disease or diabetes. This
subgroup was not separately analysed (or if it
was analysed it was not reported).
• This study provides no evidence concerning
the benefits treatment for this subgroup
Group 3
• Amongst people similar to the study population
there is no need to test blood cholesterol levels
prior to commencing a statin.
• Once a decision has been taken to start statin
treatment there is no need to monitor blood
cholesterol levels.
• Patients with Total Cholesterol <3.5 mmol/L were
excluded.
• Outcome event rate benefit from simvastatin
40mg did not vary with starting cholesterol level
or pre-randomisation LDL response.
• But the study does not tell us whether much
larger reductions in LDL cholesterol would give
rise to larger reductions in outcome event rates.
• This study provides no evidence of need to check
cholesterol levels during treatment with
simvastatin 40mg.
• This information needs to be taken in the context
that other studies have reported greater benefit
for greater reduction in cholesterol levels, and
QOF provides a financial incentive to treat to a
target level.
Group 4
• Statins should be stopped at age 80.
• Women benefit from statin treatment to the
same extent as men.
• Once started statin treatment should be
continued indefinitely.
Trial population were aged 40 to 80 at outset – so the
oldest were 85 at end of trial.
Event rate
<65
65 – 69
>70
Simva
16.9%
20.9%
23.6%
Placebo
22.1%
27.2%
28.7%
ARR
5.2%
6.3%
5.1%
NNT
19
16
20
Considerable overlap of confidence intervals of event
rate ratios for different age subgroups suggests no
statistically significant difference in treatment effect
with age within the age groups studied.
Trend chi2 0.73 (nb 3.84 ~ p<0.05)
Note: 67% of trial participants were male
Event rate
Male
Female
Simva
21.6%
14.4%
Placebo
27.6%
17.7%
ARR
6.0%
3.3%
NNT
17
30
Note the pattern of overlap of confidence intervals
Heterogeneity chi2 0.76
The actual results tell us that there is a difference in benefit
between male and female, but the statistical tests tell us that
this difference is not significant.
• Study duration was 5 years, so strictly it only
informs us of benefits and harms over the first
5 years of treatment.
• But figures 5 & 6 give some information which
allows us to predict that benefits would
continue for longer than 5 years – how much
longer is a matter of opinion / judgment.
Event
rate
Year
1
2
3
4
5+
simva
4.7%
3.9%
3.9%
3.8%
5.8%
placebo
5.1%
5.6%
5.6%
5.2%
7.3%
ARR
0.4%
1.7%
1.7%
1.4%
1.5%
NNT
250
59
59
71
67
Event
rate
Year
ARR
1
2
3
4
5+
0.4%
1.7%
1.7%
1.4%
1.5%
Diff in
Adjusted
Statin
use
85%
76%
67%
59%
50%
ARR
NNT
0.5%
2.2%
2.5%
2.4%
3.0%
200
45
40
42
33
• Before we can apply the results of any individual
primary study we have to place it in the context
of all other relevant research.
• > Systematic Reviews, Guidelines
• What is already known
• What this study adds
• Has anything been added since this study?
Levels of Evidence:
I Systematic Review of all relevant RCT’s
II At least one good quality RCT
IIIGood non-randomised trials; cohort or casecontrol studies.
IV‘Expert’ opinion
Further Reading
• Cochrane handbook for systematic reviews of
interventions
http://www.cochrane.org/resources/handbook/Handbook4.2.6Sep2006.pdf
• Consort Statements
http://www.consort-statement.org/?o=1001
• Bandolier
• BMJ
Odds Ratios
Odds of an event = no. of events / no. of non-events
e.g. 51 boys/100 births
Odds of boy = 51/49 = 1.04
Odds >1 means event more likely to happen than not
Odds of an impossibility are zero
Odds of a certainty are infinity
Odds ratio = odds in intervention group /
odds in control group
Odds Ratios (cont.)
When events are rare Odds and Risk are similar
OR and RR are similar
As prevalence (control event rate) and
OR increase the error in using OR as an
approximation for RR becomes unacceptable.
In HPS:
Simvastatin
Placebo
Event
2033
2585
No Event
8236
7684
10269
10267
In the simvastatin group:
the odds of an event are 2033/8236 = 0.247 or 24.7%
the risk of an event is 2033/10269 = 0.198 or 19.8%
In the placebo group:
The odds of an event are 2585/7684 = 0.336 or 33.6%
The risk of an event is 2585/10267 = 0.252 or 25.2%
The odds ratio (=relative odds) is 0.247/0.336 = 0.735 or 73.5%
The risk ratio (=relative risk) is 0.198/0.252 = 0.786 or 78.6%
An absolute difference of 51:1000 and a relative error of ~7%
Odds ratios are used because of their
superior mathematical properties:
• They can always take values between 0 and infinity;
values RR can take are dependent on CER.
• With OR’s the relationship between the two possible
outcomes (event or not event) is reciprocal – not so for
RR.
• OR’s always used in case-control studies where disease
prevalence is not known.
• When adjusting for confounding factors which affect
event rates, logistic regression models (the correct
approach) use odds and report effects as odds ratios.