Transcript Identifying Study Biases

Rapid appraisal of the
literature:
Identifying study biases
Rita Popat, PhD
Clinical Assistant Professor
Division of Epidemiology
Stanford University School of Medicine
August 7, 2007
What is critical appraisal?
 Balanced assessment of benefits and
strengths of research against its flaws and
weaknesses
 Assessment of research process and
results
 To be undertaken by all health
professionals as part of their work
Why should we critically appraise?
 Published research is not always valid – we cannot take
conclusions for granted, even if the article is published
in a peer-reviewed journal.
 Published research is not always relevant – the abstract
may indicate relevance but you will need to read the
complete article to judge its applicability to your own
practice/circumstances.
 To improve clinical effectiveness, we need a systematic
framework to interpret research, rather than relying on a
haphazard or casual approach.
3
Key Steps To Effective Critical
Appraisal
1. Are the results valid?
Focus of today’s
lecture
2. What are the results?
3. How will these results help me work
with my patients?
4
Outline
 Quick review of study designs
 What is validity?
 Identifying study biases that can threaten
internal validity
5
Observational vs. Experimental
Studies
 Experimental studies – the investigator
tries to control the environment in which the
hypothesis is tested (the randomized,
double-blind clinical trial is the gold standard)
 Observational studies – the population is
observed without any interference by the
investigator
6
Why Observational Studies?





Cheaper
Faster
Can examine long-term effects
Hypothesis-generating
Sometimes, experimental studies are not
ethical (e.g., randomizing subjects to smoke)
 Sometimes, experimental studies are not
possible – examples…
 randomizing subjects to gestational
diabetes
 studying natural progression of a disease
 studying long term effects of drugs
7
THE LANCET • Vol 359 • January 5, 2002
Cohort Studies
Disease (IE+)
Target
population
Exposed
(E+)
Disease-free
cohort
Not
Exposed
(E-)
Disease-free
Disease (IE-)
Disease-free
TIME
9
Measure of association in cohort studies:
Relative Risk (RR)
Exposure +
Exposure -
Outcome
+
a
Outcome
b
c
a
I E
a

b
RR 

c
I E
cd
d
a+b
c+d
Incidence
(probability) of
outcome among
exposed
Incidence
(probability)of
outcome among
unexposed
10
Active †
Inactive
T2DM
+
376
T2DM
985
376
0.029
12936
RR 

 0.73
985
0.0395
24942
12560
12936
23957
24942
Interpretation: Active women
are 27% less likely to
develop T2DM compared to
inactive women
†Energy expenditure was at least 1000 kcal/wk.
11
Cohort Studies: Advantages & Disadvantages
Advantages
 Allows you to measure true rates and risks of disease
for the exposed and the unexposed groups.
 Temporality is correct
 Can be used to study multiple outcomes.
 Prevents bias in the ascertainment of exposure that
may occur after a person develops a disease.
Disadvantages
 Can be lengthy and costly!
 Loss to follow-up is a problem (especially if nonrandom).
12
Case-Control Studies
Cases
Exposed in
past
(outcome +)
Not exposed
Target
population
Controls
Exposed in
the past
(outcome -)
Not Exposed
13
Measure of Association in
case-control studies: Odds Ratio (OR)
Exposure (E+)
Disease (D+)
Cases
a
No disease(D-)
Controls
b
c
d
No exposure(E-)
a+c
OR =
P ( E| D )
P ( E | D )
Odds of exposure
among cases
Odds of exposure
among controls
b+d
=
P ( E|D  )
P ( E | D  )
=
a a c
c a c
b b d
d b d
=
ad
bc
14
Cases
Controls
NSAIDS+
561
971
NSAIDS-
71
74
(561)(74)
OR 
 0.60
(71)(971)
Interpretation: NSAIDs use
is associated with a 40%
reduction in the risk of colon
cancer
15
Case-Control Studies:
Advantages & Disadvantages
 Advantages:
• Cheap and fast
• Great for rare diseases
 Disadvantages:
• Exposure estimates are subject to
• recall bias (those with the disease are searching for
reasons why they got sick and may be more likely to
report an exposure)
• interviewer bias (interviewer may prompt a positive
response in cases).
• Temporality is a problem (did exposure cause
disease or disease cause exposure?)
16
Intervention Studies
Disease
Intervention
group
Target
population
Disease-free
cohort
No
intervention
group
Disease-free
Disease
Disease-free
TIME
17
Eligible participants
Randomized
Standard lifestyle recommendations
Intensive
Lifestyle
(n = 1079)
Metformin
(n = 1073)
Placebo
(n = 1082)
18
DPP trial: Primary Outcome - Diabetes
19
Intervention Studies:
Advantages & Disadvantages
 Advantages:
• Allows randomization (controls for confounding)
• Allows double-blind assessment (controls bias)
 Disadvantages:
• Can be lengthy and costly!
• Loss to follow-up is a problem (especially if nonrandom).
• Ethical limitations
20
Causation in human health & disease
 Association does not prove causation
• If a putative risk factor and the occurrence of an
outcome are strongly associated with each other it
does not provide evidence that the risk factor
causes the disease, only implies that it is
correlated with outcome
• Non-causal explanations may cause a spurious
association – study biases (measurement error,
selection bias, confounding, sampling error)
21
Study validity
 INTERNAL VALIDITY
Do we believe the results?
 EXTERNAL VALIDITY
Can the results be applied to the target
population i.e., beyond the subjects in
the study?
22
Threats to Internal validity:
Non-causal explanations due to study
biases
 Confounding
 Selection bias
 Misclassification bias (measurement error)
23
Confounding
Confounding is the defined as a distortion of
an exposure-outcome association brought
about by the association of another factor
with both the outcome and exposure
exposure
outcome
Confounding
factor
24
JAMA. 2004;292:1188-1194
Research question - Is physical activity associated with
risk of T2DM?
Physical
activity
T2DM
+
BMI
Potential confounder
+
25
Why worry about confounding?
 Spurious association
 Exaggerate an association (over-estimate)
 Attenuate an association (under-estimate)
 Obscure an association
26
Methods for controlling confounding
 Design phase
• Randomization
• Matching
• Restriction
 Analysis phase: statistical adjustment for
confounders
27
Controlling confounding by randomization
Identifying potential confounders
Subset of Table 1 from JAMA. 2004;292:1188-1194
Statistical adjustment for confounding
Crude HR =2.9/3.9 = 0.74
Weinstein et al. JAMA 292:1188-94
Is BMI a confounder of the relationship b/w physical activity and T2DM?
Compare the crude hazard ratio (HR) to the adjusted HR.
Statistical adjustment for confounding
Weinstein et al. JAMA 292:1188-94
10% rule for identifying confounders:
(Crude HR - Adjusted HR) X 100  10%
Crude HR
True or false
A randomized clinical trial design cannot be
affected by bias due to confounding?
 Answer: False (if randomization is not done
appropriately, then can introduce bias due to
confounding)
32
Identifying bias due to confounding in
a RCT
 Check the randomization procedure in the
methods section (e.g., blocked randomization
schemes when sample size is small)
 Check Table 1 see if groups are balanced
 If not, how was it handled?
 Was intention to treat analysis used?
33
JAMA. 2006;296:2441-2450
JAMA. 2006;296:2441-2450
Selection bias
 A form of sampling bias due to systematic
differences between those who are selected
for study (or agree to participate) and those
who are not selected (or refuse to
participate).
36
Selection bias
 Improper selection of cases or controls in a casecontrol study
 Subjects lost to follow-up varies according to both
the exposure of interest and the outcome (e.g., in
prospective cohort studies and clinical trials)
 Can affect any study design, although case-control
studies more prone to selection bias
 Selection bias can cause either overestimates or
underestimates of the true associations between
the exposure and disease in the underlying
population
37
Identifying Selection bias
 What are the response rates?
 Was follow-up complete?
 Ideally should have follow-up for at least 80% of
the initial sample/cohort
 Does drop out differ in the groups being compared
(e.g., treatment and placebo groups)?
38
Selection bias example:
In this case-control study, response rate was ~80% in
cases and ~60% in controls.
Could there be selection bias…especially among
controls?
39
Selection bias example:
Scenario: NSAID users in the base population were more
likely to participate in this study than were non-NSAID users
p1
p2
 So observed proportion of NSAID users among controls
(92.9%) is greater than true proportion of exposure
 Hence observed odds ratio of 0.54 is an overestimate (i.e.,
true odds ratio is greater than 0.54)
 [Note that Odds ratio = p1* (1- p2) / p2 *(1- p1)]
40
Some strategies for minimizing
selection bias
 Careful enumeration and thorough attempts at
recruiting all cases within the source population
 High standards for methods of control selection
(population-based ideal)
 Minimizing non-response and refusals
 Minimizing loss to follow-up (in cohort and RCTs)
41
Information bias (aka measurement error)
 Misclassification of outcome
 Misclassification of exposure status
42
Information bias (measurement error)
Imperfect definitions of study variables
(outcome or predictors) or flawed data
collection procedures
Erroneous classification of
– outcome
– exposure
43
Information bias
misclassification
Gold-standard
Outcome
+
Outcome
-
Outcome +
a
TP
b
FP
a+b
Outcome -
c
FN
d
TN
c+d
a+c
b+d
Study
Sensitivity = a / (a+c)
Specificity = d / (b+d)
44
Misclassification of the outcome
Exposure +
(treatment)
Exposure –
(placebo)
Outcome +
a
Outcome b
c
d
 Non-differential misclassification occurs when the
degree of misclassification of outcome is independent
of exposure status
 Tends to bias the association toward the null
 Occurs when the sensitivity and specificity of the
classification of outcome are same for exposed and
non-exposed groups but less than 100%
45
Misclassification of the outcome
Exposure +
(treatment)
Exposure –
(placebo)
Outcome +
a
Outcome b
c
d
Differential misclassification occurs when the degree of
misclassification differs between the groups being
compared.
 May bias the association either toward or away from
the null hypothesis
 Occurs when the sensitivity and specificity of the
classification of outcome differ for exposed and nonexposed groups
46
Misclassification of the outcome:
NSAIDs and colon cancer example
Cases
Controls
NSAIDS+
561
971
NSAIDS-
71
74
Most likely scenario in this study
Is misclassification of outcome likely to be nondifferential or differential with respect to exposure?
47
JAMA. 2006;296:2441-2450
Study Measures
The primary measures were the Medical Outcomes Study 36Item Short- Form Health Survey (SF-36) bodily pain and
physical function scales and the American Academy of
Orthopaedic Surgeons MODEMS version of the Oswestry
Disability Index (ODI).
Secondary measures included patient self-reported
improvement, work status, and satisfaction with current
symptoms and with care. Symptom severity was measured by
the Sciatica Bothersomeness Index (range, 0-24; higher scores
represent worse symptoms).
Is misclassification of outcome likely to be nondifferential or differential with respect to exposure?
Misclassification of the exposure
Exposure +
Cases
(outcome +)
a
Controls
(outcome -)
b
Exposure -
c
d
 Non-differential misclassification occurs when the
degree of misclassification of exposure is
independent of outcome/disease status
 Tends to bias the association toward the null
 Occurs when the sensitivity and specificity of the
classification of exposure are same for those with and
without the outcome but less than 100%
50
Misclassification of the outcome
Exposure +
Cases
(outcome +)
a
Controls
(outcome -)
b
Exposure -
c
d
Differential misclassification occurs when the degree of
misclassification differs between the groups being
compared.
 May bias the association either toward or away from the
null hypothesis
 Occurs when the sensitivity and specificity of the
classification of exposure differ for those with and without
the outcome
51
Misclassification of the exposure:
NSAIDs and colon cancer example
Cases
Controls
NSAIDS+
561
971
NSAIDS-
71
74
Non-differential: poor recall in cases and controls
Differential: cases recall NSAID use better than controls?
Is misclassification of exposure likely to be nondifferential or differential with respect to outcome?
52
Some strategies for minimizing
misclassification bias
 Test reliability and validity of instruments used to
determine outcome and exposure
 Similar methods for determining outcome and exposure in
all study subjects
 Train interviewers, blind interviewers to outcome status
and study hypothesis (  interviewer bias)
 Blind subjects to study hypothesis ( recall bias)
 Use incident cases in case-control studies, not prevalent
( recall bias)
 Try to use objective measures (e.g., pharmacy records
vs. self-report use of medications)
53
Summary : Study designs and biases
Threats to
internal validity
Case-control
Cohort
RCT
Confounding
Generally present
Generally present
Not likely (due to
randomization)
Selection bias
Likely (e.g., when
low response
rates)
May occur due to
differential loss to
follow up
May occur due to
differential loss to
follow-up
Misclassification
of exposure
More likely to be
differential
Generally nondifferential
Not likely; if exists
then nondifferential (e.g.,
drop-in/drop out)
Misclassification
of outcome
Most likely nondifferential
Most likely nondifferential
Most likely nondifferential
54
Once you are satisfied the study findings
are valid…
You can now ask whether the association
causal?
Evaluate positive features of causation
 Temporality
 Strength of the association
 Dose-response relationship
 Consistency of findings
 Biologic plausibility
55
Best
Experiment/RCT
Prospective cohort
Cost and
ease
Retrospective cohort
Case-control
Correlation/x-section
Proof
of cause
Case series
Case report
Best
56
Critical appraisal tools
 Assist with systematic critique of published
research papers
 Several tools available
 One of my favorite…
http://www.muhc-ebn.mcgill.ca/EBN_tools.htm#evidence
and look for link to “CASP Appraisal Tools”
57
Critical appraisal of an article about
therapy or prevention
Primary guides:
 Was the assignment of patients to treatment groups
randomized?
 Were all patients who entered the trial properly accounted
for at the conclusion?
 Was follow up complete?
 Were patients analyzed in the groups to which they were
randomized, that is, was an intention to treat analysis
used?
58
Critical appraisal of an article about
therapy or prevention
Secondary guides:
 Were patients, health workers and study personnel blinded to
treatment?
 Does the study provide evidence that blinding was effective?
 Were the groups similar at the start of the trial?
 Was there an adequate Table 1?
 If not, were adjustments made for differences?
 Aside from the experimental intervention, were the groups
treated equally?
59
Critical appraisal of an article about
therapy or prevention





What were the results?
How large was the treatment effect?
How precise was the estimate of the treatment effect?
Were confidence intervals given?
Will the results help me with my patients?
Will discuss these aspects on August 9th, 2007!
60