Transcript Document 7276745

Sources of Bias in
Randomised Controlled Trials
David Torgerson
Director, York Trials Unit
[email protected]
www.rcts.org
Selection Bias - A reminder
• Selection bias is one of the main threats to
the internal validity of an experiment.
• Selection bias occurs when participants
are SELECTED for an intervention on the
basis of a variable that is associated with
outcome.
• Randomisation or other similar methods
abolishes selection bias.
After Randomisation
• Once we have randomised participants we
eliminate selection bias but the validity of
the experiment can be threatened by other
forms of bias, which we must guard
against.
Forms of Bias
•
•
•
•
•
•
•
Subversion Bias
Technical Bias
Attrition Bias
Consent Bias
Ascertainment Bias
Dilution Bias
Recruitment Bias
Bias (cont)
•
•
•
•
•
Resentful demoralisation
Delay Bias
Chance Bias
Hawthorne effect
Analytical Bias.
Subversion Bias
• Subversion Bias occurs when a
researcher or clinician manipulates
participant recruitment such that groups
formed at baseline are NOT equivalent.
• Anecdotal, or qualitative evidence (I.e
gossip), suggest that this is a widespread
phenomenon.
• Statistically this has been demonstrated as
having occurred widely.
Subversion - qualitative
evidence
• Schulz has described, anecdotally, a
number of incidents of researchers
subverting allocation by looking at sealed
envelopes through x-ray lights.
• Researchers have confessed to breaking
open filing cabinets to obtain the
randomisation code.
Schulz JAMA 1995;274:1456.
Quantitative Evidence
• Trials with adequate concealed allocation
show different effect sizes, which would
not happen if allocation wasn’t being
subverted.
• Trials using simple randomisation are too
equivalent for it to have occurred by
chance.
Poor concealment
• Schulz et al. Examined 250 RCTs and
classified them into having adequate
concealment (where subversion was
difficult), unclear, or inadequate where
subversion was able to take place.
• They found that badly concealed allocation
led to increased effect sizes – showing
CHEATING by researchers.
Comparison of concealment
Allocation
Concealment
Adequate
Effect Size
OR
1.0
Unclear
0.67
Inadequate
0.59
Schulz et al. JAMA 1995;273:408.
P < 0.01
Case Study
• Subversion is rarely reported for individual
studies.
• One study where it has been reported was
for a large, multicentred surgical trial.
• Participants were being randomised to 5+
centres using sealed envelopes.
Case-study (cont)
• After several hundred participants had
been allocated the study statistician
noticed that there was an imbalance in
age.
• This age imbalance was occurring in 3 out
of the 5 centres.
• Independently 3 clinical researchers were
subverting the allocation
Mean ages of groups
Clinician
Experimental
Control
All p < 0.01
59
63
1 p =.84
62
61
2 p = 0.60
43
52
3 p < 0.01
57
72
4 p < 0.001
33
69
5 p = 0.03
47
72
Others p = 0.99
64
59
Envelope Number
Example of Subversion
30
25
20
15
10
5
0
1
3
5
7
11 13 15 17 19
Recruitment Sequence
9
21
23
25
Concealment
• Both the Schulz and Kjaergard considered
sealed opaque envelopes to be ‘adequate’
measures of concealment.
• Envelopes can be subverted by being
opened in advance.
More Evidence
• Hewitt and colleagues examined the association
between p values and adequate concealment in
4 major medical journals.
• Inadequate concealment largely used opaque
envelopes.
• The average p value for inadequately concealed
trials was 0.022 compared with 0.052 for
adequate trials (test for difference p = 0.045).
Hewitt et al. BMJ;2005: March 10th.
More Examples
• Berger has collected 30 case examples of
potential subversion of the allocation
process in clinical trials.
• Because allocation subversion is scientific
misconduct it is likely that there are many
other, undetected, cases.
Berger. Selection Bias and Covariate Imbalances in Randomized
Clinical Trials 2005: Wiley, Chicester.
Recent Blocked Trial
“This was a block randomised study (four patients
to each block) with separate randomisation at
each of the three centres. Blocks of four cards
were produced, each containing two cards
marked with "nurse" and two marked with
"house officer." Each card was placed into an
opaque envelope and the envelope sealed. The
block was shuffled and, after shuffling, was
placed in a box.”
Kinley et al., BMJ 2002 325:1323.
What is wrong here?
Southampton
Sheffield
Doncaster
Doctor Nurse
Doctor Nurse
Doctor Nurse
500
308
118
511
Kinley et al., BMJ 325:1323.
319
118
Problem?
• If block randomisation of 4 were used then
each centre should not be different by
more than 2 patients in terms of group
sizes.
• Two centres had a numerical disparity of
11. Either blocks of 4 were not used or the
sequence was not followed.
Restricted allocation and
subversion
• The drawback with any form of allocation
restriction is that it allows some prediction.
• Simple randomisation has no ‘memory’ of the
previous allocation. In contrast, blocked
allocation allows the probability of an allocation
to be linked to the previous allocation.
• Merely guessing that the next allocation will be
the opposite of the previous one will result in a
prediction more accurate than by chance.
• This can, in theory, allow subversion.
Possible subversion
• In a RCT of rehabilitation for the treatment of hip
fracture gross baseline imbalances were
detected favouring the control group.
• Secure telephone allocation had been used. But
blocked allocation, size 6, had been used.
• Exploratory analysis of imbalances suggested
partially successful prediction of block allocation.
Turner J. 2002, Unpublished PhD Thesis, University of York.
Wither restricted allocation?
• Simple randomisation followed by analysis
of covariance (ANCOVA) is as efficient as
restricted randomisation and ANCOVA for
sample sizes > 50.
• Restricted allocation increases risk of
prediction and predictability.
• For large trials simple allocation followed
by ANCOVA reduces risk of prediction.
Rosenberger WF, Lachin JM. Randomisation in clinical trials: Theory and
practice. Wiley Interscience, 2002, John Wiley and Sons, New York.
Subversion - more evidence
• In a survey of 25 researchers 4 admitted to
keeping ‘a log’ of previous allocations to try and
predict future allocations.
Brown et al. Stats in Medicine, 2005,24:3715.
Testing for subversion
• Comparison of baseline characteristics
may help if subversion is suspected.
Although this will only identify gross
subversion.
• If blocked allocation is used a statistical
test – Bergner-Exner test, may help
identify subversion.
Concealment: Recommendations
• Allocation sequence must be
independently generated and kept secret
from the people who are enrolling
participants.
• A secure method of giving allocation to the
recruiters must be developed, opaque
envelopes are inadequate.
Subversion - summary
• Appears to be widespread.
• Secure allocation usually prevents this
form of bias.
• Need not be too expensive.
• Essential to prevent cheating.
Secure allocation
• Can be achieved using telephone
allocation from a dedicated unit.
• Can be achieved using independent
person to undertake allocation.
Technical Bias
• This occurs when the allocation system
breaks down often due a computer fault.
• A great example is the COMET I trial
(COMET II was done because COMET 1
suffered bias).
COMET 1
• A trial of two types of epidural anaesthetics
for women in labour.
• The trial was using MIMINISATION via a
computer programme.
• The groups were minimised on age of
mother and her ethnicity.
• Programme had a fault.
COMET Lancet 2001;358:19.
COMET 1 – Technical Bias
AGE
Traditional
Combined
Low dose
Total
388
335
331
13
(3%)
179
(53%)
173
(52%)
<25 years
COMET II
• This new study had to be undertaken and
another 1000 women recruited and
randomised.
• LESSON – Always check the balance of
your groups as you go along if computer
allocation is being used.
Attrition Bias
• Usually most trials lose participants after
randomisation. This can cause bias, particularly
if attrition differs between groups.
• If a treatment has side-effects this may make
drop outs higher among the less well
participants, which can make a treatment appear
to be effective when it is not.
Attrition Bias
• We can avoid some of the problems with
attrition bias by using Intention to Treat
Analysis, where we keep as many of the
patients in the study as possible even if
they are no long ‘on treatment’.
Selection bias after randomisation
• Selection bias is avoided if ALL
participants who are randomised are
completely followed up.
• Often there is some attrition – after
randomisation some refuse to continue to
take part.
• Or some may refuse the intervention but
can still be tracked – IMPORTANT to
distinguish between these.
What is wrong here?
Random allocation
160 children 8 from
Each school
1 school 8 children withdrew
N = 17 children replaced following
discussion with teacher
76 children allocated to control
76 allocated to intervention group
Ascertainment Bias
• This occurs when the person reporting the outcome can
be biased.
• A particular problem when outcomes are not ‘objective’
and there is uncertainty as to whether an event has
occurred.
• Example, of homeopathy study of histamine, showed an
effect when researchers were not blind to the allocation
but no effect when they were.
• Multiple sclerosis treatment appeared to be effective
when clinicians unblinded but ineffective when blinded.
Resentful Demoralisation
• This can occur when participants are
randomised to treatment they do not want.
• This may lead to them reporting outcomes
badly in ‘revenge’.
• This can lead to bias.
Resentful Demoralisation
• One solution is to use a patient preference
design where only participants who are
‘indifferent’ to the treatment they receive
are allocated.
• This should remove its effects.
Hawthorne Effect
• This is an effect that occurs by being part
of the study rather than the treatment.
Interventions that require more TLC than
controls could show an effect due to the
TLC than the drug or surgical procedure.
• Placebos largely eliminate this or TLC
should be given to controls as well.
Analytical Bias
• Once a trial has been completed and data
gathered in it is still possible to arrive at
the wrong conclusions by analysing the
data incorrectly.
• Most IMPORTANT is ITT.
• Also inappropriate sub-group analyses is a
common practice.
Intention To Treat
• Main analysis of data must be by groups
as randomised. Per protocol or active
treatment analysis can lead to a biased
result.
• Those patients not taking the full treatment
are usually quite different to those that are
and restricting the analysis can lead to
bias.
Sub-Group Analyses
• Once the main analysis has been
completed it is tempting to look to see if
the effect differs by group.
» Is treatment more or less effective in women?
» Is it better or worse among older people?
» Is treatment better among people at greater
risk?
Sub-Groups
• All of these are legitimate questions. The
problem is the more subgroups one looks
at the greater is the chance of finding a
spurious effect.
• Sample size estimations and statistical
tests are based on 1 comparison only.
Sub-Group and example.
• In a large RCT of asprin for myocardial
infarction a sub-group analysis showed
that people with the star signs Gemini and
Libra aspirin was INEFFECTIVE.
• This is complete NONSENSE!
• This shows dangers of subgroup analyses.
Lancet 1988;ii:349-60.
Sub groups
• To avoid spurious findings these should be
pre-specified and based on a reasonable
hypothesis.
• Pre-specification is important avoid data
dredging as if you torture the data enough
it will confess.
Summary
• Despite the RCT being the BEST research
method unless expertly used it can lead to
biased results.
• Care must be taken to avoid as many
biases as possible.