Transcript Slide 1

Sources of bias in RCTs
With focus on blinding and
concealment of allocation
David Torgerson
Background
• Most reported educational (and health)
trials are of POOR quality.
• They are too small and do not adequately
report their methods.
All RCTs are NOT the same.
• Although the RCT is rightly regarded as
the premier research method, by those
who know, some trials are better than
others.
Selection Bias - A reminder
• Selection bias is one of the main threats to
the internal validity of an experiment.
• Selection bias occurs when participants
are SELECTED for an intervention on the
basis of a variable that is associated with
outcome.
• Randomisation or other similar methods
abolishes selection bias.
After Randomisation
• Once we have randomised participants we
eliminate selection bias but the validity of
the experiment can be threatened by other
forms of bias, which we must guard
against.
REMEMBER
• Whilst most RCTs are susceptible to a
range of potential biases other study types
(e.g., pre test post test, quasi experiments)
are also susceptible to the same biases.
Blinding
• The ‘gold-standard’ in health care trials is
supposedly the double blind placebo
controlled trial.
• It is helpful to blind some people some of
the time but may not be always
appropriate.
What does blinding achieve?
• If a participant, their doctor (teacher) and
the researcher are unaware of treatment
received. This prevents a number of
potential biases including ascertainment
and performance.
Ascertainment Bias
• This occurs when the person reporting the
outcome can be biased.
• A particular problem when outcomes are not
‘objective’ and there is uncertainty as to whether
an event has occurred.
• For example, if the person marking an essay
knows into what group the person belongs this
might affect their mark.
Example.
• A group of student’s essays were randomly
assigned photographs purporting to be the
student. The photos were of people judged to
be “attractive” “average” “below average”. The
average mark was significantly HIGHER for the
average looking student.
• Why? Markers were biased into marking higher
for students whom they believed were average
looking (like themselves).
Performance Bias
• Blinding, by sham or placebo, can reduce this
bias BUT is it a bias?
• This is where the student because they are in a
group they like ‘performs’ better than in a group
they don’t like. It could be argued that if they like
the intervention and perform better this is part of
the intervention. Conversely, if the intervention
is ineffective despite being liked better (e.g.,
computers) then it really is ineffective.
Preference effects
• When students (or patients) prefer an
intervention this may bias an outcome.
One way of attempting to control for this
bias is to ask before randomisation the
student’s preference and use this in the
analysis.
Subversion Bias
• Subversion Bias occurs when a researcher or
clinician manipulates participant recruitment
such that groups formed at baseline are NOT
equivalent.
• Anecdotal, or qualitative evidence (I.e gossip),
suggest that this is a widespread phenomenon.
• Statistically this has been demonstrated as
having occurred widely.
Subversion - qualitative
evidence
• Schulz has described, anecdotally, a
number of incidents of researchers
subverting allocation by looking at sealed
envelopes through x-ray lights.
• Researchers have confessed to breaking
open filing cabinets to obtain the
randomisation code.
Schulz JAMA 1995;274:1456.
Subversion in a non health trial
• Boruch describes a trial in USA where
incidents of domestic violence were being
randomised to a ‘caution’ or being seen in
the police station.
• Some evidence that police officers made
‘sure’ that offenders they knew got taken
to the police station.
• This WILL damage the trial.
Subversion and Education
• I don’t know of an educational trial that has
been ‘subverted’ BUT they almost certainly
exist.
• Educational researchers are no different
from health researchers and are just as
likely to subvert their trials as we do!
Quantitative Evidence
• Trials with adequate concealed allocation show
different effect sizes, which would not happen if
allocation wasn’t being subverted.
• Trials using simple randomisation are too
equivalent for it to have occurred by chance.
• Often educational trials used ‘paired’
randomisation but have UNEQUAL numbers in
each group – which means they lost one some
where!
Poor concealment
• Schulz et al. Examined 250 RCTs and
classified them into having adequate
concealment (where subversion was
difficult), unclear, or inadequate where
subversion was able to take place.
• They found that badly concealed allocation
led to increased effect sizes – showing
CHEATING by researchers.
Comparison of concealment
Allocation
Concealment
Adequate
Effect Size
OR
1.0
Unclear
0.67
Inadequate
0.59
Schulz et al. JAMA 1995;273:408.
P < 0.01
0
.05
Density
.1
.15
Recent Health Trials.
-10
-5
0
logit (p-value)
Adequate
Unclear
Hewitt et al. 2004; submitted
Inadequate
5
Case Study
• Subversion is rarely reported for individual
studies.
• One study where it has been reported was
for a large, multicentred surgical trial.
• Participants were being randomised to 5+
centres using sealed envelopes.
Case-study (cont)
• After several hundred participants had
been allocated the study statistician
noticed that there was an imbalance in
age.
• This age imbalance was occurring in 3 out
of the 5 centres.
• Independently 3 clinical researchers were
subverting the allocation
Mean ages of groups
Clinician
Experimental
Control
All p < 0.01
59
63
1 p =.84
62
61
2 p = 0.60
43
52
3 p < 0.01
57
72
4 p < 0.001
33
69
5 p = 0.03
47
72
Others p = 0.99
64
59
Envelope Number
Example of Subversion
30
25
20
15
10
5
0
1
3
5
7
11 13 15 17 19
Recruitment Sequence
9
21
23
25
Using Telephone Allocation
Clinician
Experimental
Control
All p = 0.37
59
57
1 p =.62
57
57
2 p = 0.24
60
51
3 NA
61
70
4 p =0.99
63
65
5 p = 0.91
57
62
Others p = 0.99
59
56
Subversion - summary
• Appears to be widespread in health trials
almost certainly occurs in psychology trials
but remains undetected.
• Secure allocation usually prevents this
form of bias.
• Need not be too expensive.
• Essential to prevent cheating.
Secure allocation
• Can be achieved using telephone
allocation from a dedicated unit.
• Can be achieved using independent
person to undertake allocation.
Attrition Bias
• Usually most trials lose participants after
randomisation. This can cause bias, particularly
if attrition differs between groups.
• If a treatment has side-effects this may make
drop outs higher among the less well
participants, which can make a treatment appear
to be effective when it is not.
Attrition Bias
• We can avoid some of the problems with
attrition bias by using Intention to Teach
(or treat) Analysis, where we keep as
many of the patients in the study as
possible even if they are no long ‘on
treatment’.
Dilution Bias
• This occurs when the intervention or
control group get the opposite treatment.
• For example, in a trial of domestic violence
the judge over-rode the random
assignment and put 3.5% (14) men in the
intervention group.
Feder & Dugan, Justice Quarterly, 2002;19:343.
Resentful Demoralisation
• This can occur when participants are
randomised to treatment they do not want.
• This may lead to them reporting outcomes
badly in ‘revenge’.
• This can lead to bias.
Resentful Demoralisation
• One solution is to use a patient preference
design where only participants who are
‘indifferent’ to the treatment they receive
are allocated.
• This should remove its effects.
Hawthorne Effect
• This is an effect that occurs by being part
of the study rather than the treatment.
Interventions that require more TLC than
controls could show an effect due to the
TLC than the drug or surgical procedure.
• TLC should be given to controls as well.
Forms of Bias
•
•
•
•
•
•
•
Subversion Bias
Technical Bias
Attrition Bias
Consent Bias
Ascertainment Bias
Dilution Bias
Recruitment Bias
Bias (cont)
•
•
•
•
•
Resentful demoralisation
Delay Bias
Chance Bias
Hawthorne effect
Analytical Bias.
Conclusions
• There are a range of biases associated
with POORLY designed RCTs.
• Many, if not all, can be eliminated through
careful attention to methods.