Epidemiology:
Download
Report
Transcript Epidemiology:
Randomized Controlled Trials:
Allocation Concealment
Kenneth F. Schulz, PhD
Fixation on Sequence Generation
Traditionally, many medical researchers
mistakenly consider the sequence
generation process as “randomization”
They frequently ignore allocation
concealment
Without adequate allocation concealment,
proper random sequences can be subverted
For example, a properly generated sequence
posted on a bulletin board
Allocation Concealment
Prevents selection bias by
concealing the allocation sequence
from those assigning participants to
intervention groups
Until the moment of assignment
Mechanisms for Selection Bias
in RCTs
If those responsible for entering
participants know, or can detect, the
upcoming treatment allocations:
Decide entrance based on that knowledge
Channel participants with a better prognosis
to the experimental group and those with a
poorer prognosis to the control group, or vice
versa
In any case, they introduce bias into the
treatment comparison
Do Not Confuse Allocation
Concealment with Blinding
Allocation concealment seeks to prevent
selection bias, protects assignment
sequence before and until allocation, and
can always be successfully implemented
In contrast, blinding seeks to prevent
ascertainment bias, protects sequence
after allocation, and cannot always be
successfully implemented
Importance of Allocation
Concealment
Unclearly concealed and inadequately
concealed trials, compared to
adequately concealed trials,
exaggerated the estimates of an
intervention’s effectiveness by 30% to
40%, on average
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical
evidence of bias: dimensions of methodological quality
associated with estimates of treatment effects in controlled
trials. JAMA 1995;273:408-412.
Similar Results on Allocation
Concealment in a Separate Study
Replicated our methods in different
subject areas (digestive & circulatory
diseases and mental health)
Examined 11 meta-analyses
Inadequately concealed trials exaggerated
estimates of effectiveness by 37%
[Moher D, et al. Does quality of reports of randomised trials
affect estimates of intervention efficacy reported in metaanalyses? Lancet 1998; 352: 609-13.]
Allocation Concealment: Summary
Adequate allocation concealment emerges
from our analyses as crucial to reducing
bias
Without it, the whole point of
randomisation vanishes and bias may
distort results
Our results support Mosteller and his
colleagues
“When the randomization leaks, the trial’s
guarantee of lack of bias runs down the drain”
Method of allocation of treatment in 208
controlled trials in head injury
Dickinson K, et al., BMJ 2000;320:1308-1311.
Method of allocation
No. of trials
Adequate
Centralised randomisation by telephone
Numbered/coded identical containers administered sequentially
1
11
Randomisation scheme controlled by pharmacy
8
Sequentially numbered, sealed, opaque envelopes
2
Not adequate
Other
18
Date of birth
1
Day of week
3
Alternation
3
Not stated
161
Actual allocation method with the
method just specified as “random”
Difficult (Peto and I. Chalmers experience)
MA of OCs
David G. knew the PI who put him in
contact with the statistician
Described a “great” method . . . Post-it
notes!
Seemed indignant . . . We asked for clarification
Perfect also for stratification—different colors!!
Personal Accounts of Deciphering
Assignment Sequences
Conducted over 20 epidemiological
workshops for medical residents and
medical junior faculty
Each included 20-25 participants
Asked how many of the participants had
deciphered, or had witnessed someone
else decipher, an assignment sequence
Personal Accounts of Deciphering
Assignment Sequences (cont.)
With assurance of anonymity, more
than half related at least one
instance of deciphering
Not half of all trials
No accurate denominator
Nevertheless, not a rare occurrence
Decipherings Run the Gamut
From Simple to Intricate
Simple the most frequent and usually
take advantage of inadequate
allocation concealment schemes:
Posting sequence on a bulletin board
Opening unsealed envelopes
Translucent envelopes to a light bulb
Opening many envelopes that were not
sequentially numbered
Intricate Decipherings Less Frequent
Needed to circumvent more adequate
allocation schemes
With sequentially numbered drug
containers
Based on appearance of tablets in
unsealed containers
Appearance of the label
With central allocation, obtaining the
next few allocations at once
Envelopes to a “hot light” in radiology
Attempted to Decipher a Numbered
Container Scheme
Gave up
Found attending physician rifling
the P.I.’s files for the assignment
sequence
Horrified? No, impressed with his
brilliance and proceeded to help
Should be kept in a locked location
Ignorance Is Not Bliss
“At the annual meeting of the Society for
Clinical Trials, held in Houston, … Schulz
reported data showing that the experts’ fears
about ignorance are justified. Schulz
examined 250 reports of clinical trials and
came up with an intriguing finding…”
[Rachel Nowak. Problems in clinical trials go far beyond misconduct.
Science 1994; 264: 1539.]
Ignorance Is Not Bliss
“Most physicians are not trained in
basic scientific principles, let alone
clinical trials”
Stephen George, Duke University, Chair of the Statistics
Committee for NCI, NIH Cancer Clinical Cooperative Groups
“I fantasize that in the future (training)
will become a requirement for running
clinical trials”
John Gallin, Director of NIH’s Clinical Center
Rigorous Trials Annoy Humans
Investigators
Certain Ps to benefit
May want the results of study to reveal
the “truth”
Proper trial procedures attempt to
impede human inclinations
Hence, aspects of properly conducted
RCTs annoy investigators
Without Methodological Rigour
The challenge of deciphering may frequently
become too great a temptation to resist
Even without intent to bias
“The only way to get rid of a temptation is to
yield to it.” Oscar Wilde
Deciphering may just reflect human
inquisitiveness and ingenuity
Does not necessarily indicate scientific
malevolence
Without Methodological Rigour
Or deciphering may reflect deliberate
acts to alter findings
Whatever the motivations, innocent
or naïve or deliberate, those actions
undermine the validity of the trial
Investigators must devote diligent,
persistent attention to randomization
Minimal Standards: Common
Allocation Concealment Approaches
Sequentially numbered, opaque,
sealed envelopes (SNOSE)
Pharmacy control
Numbered or coded containers
Central randomization
Realistically, these standards
should be exceeded
Envelopes for Allocation
Concealment
More susceptible to manipulation
through human ingenuity
Less than ideal method of allocation
concealment
If used, investigators must diligently
develop and monitor the process
Envelopes (Cont.)
SNOSE
Ensure that the envelopes are opened
sequentially
only after the P’s name
and other details are
written on the appropriate
envelope
Envelopes (Cont’d)
Pressure-sensitive or carbon paper
inside
Supports enrolment
Encourages orderly opening
Promotes proper assignment
Creates valuable audit trail
Cardboard or aluminum foil inside
envelope
“Pharmacy Controlled”
Compliance with randomization/allocation
concealment methods questionable
Aware of gross distortions
Pharmacy ran out of one drug . . .
Alternate assignment
Should not assume pharmacists
knowledge
Ensure they, and all research partners,
follow proper trial procedures
“Numbered Containers”
Excellent – prevents foreknowledge
Must take proper precautions
Appearance – exactly the same
Weight – exactly the same
Sound when shaken – exactly the same
Lettering – exactly the same
Odor – exactly the same
Sealed
Difference between concealment and blinding
“Central Randomization”
Frequently telephone
Fax
Email
Internet
“Computer”
Assignments only provided after
participants irrevocably enrolled
Must check procedures for leaks
Monitor process for adherence
RCTs: Anathema to the Human Spirit
Must acknowledge the vagaries of
human nature
Must establish methodological
safeguards that thwart attempts to
contaminate trials with bias
Subversions of Sequences
Conscious or subconscious?
Direct or compensatory?
Doesn’t matter
Biases the trial
Participants
Randomize
Placebo
40% Noncompliance
60%
Compliance
New Oral Drug
25% Noncompliance
75%
Compliance
Group representing the
policy of no treatment
Group representing the
policy of oral treatment
Outcome
Outcome
RCT Compared the Effectiveness of Clofibrate in
Preventing Cardiac Deaths in Men Who Had
Survived a Myocardial Infarction
Clofibrate
Placebo
5 Year mortality
20.2%
20.9%
(p = .55)
Eliminating deviates from
clofibrate
(80% adherence)
15.0%
20.9%
(p < .05)
Eliminating deviates from
both groups
15.0%
15.1%
Authors state that:
• One can justify almost any conclusion, dependent upon the analysis
chosen
• Manipulating deviates leads to severe bias
• Can you ever do so?
Exclusions After Randomization
Can introduce bias and should be
carefully scrutinized
All randomized patients should be
analyzed, and analyzed as part of the
group to which they were initially
assigned
ITT (Intention-to-treat)
Exclusions of LFU Damage
Internal Validity
Without outcomes from those lost to
follow-up, investigators have little
choice but to exclude them from the
analysis
Any losses damage internal validity
However, differential rates of loss among
comparison groups cause major damage
Investigators must minimize their
losses to follow-up
Retention of Trial Participants
Minimizing losses exudes difficulties
Eliminating losses may be impossible
But investigators too frequently profess
insurmountable difficulties
Most should work harder to obtain
higher follow-up rates
Investigators must commit adequate
attention and resources to develop and
implement procedures to minimize
losses
Innovative Twists that Cultivate
High Follow-up Rates
Establish many conveniently placed FU
facilities
Too often investigators expect participants to
visit a single, inconvenient location
Shortening data collection form caters to the
participants’ wishes
Long sessions lead them to vote with their feet
Be considerate
Foster follow-up by not overburdening
Also may engender higher quality data on the
main items
END
Meta-analysis of Polyglycolic
Meta-analysis of polyglycolic acid versus
catgut for perineal repair on short term pain
1 adequately concealed trial … OR = .89
4 unclearly concealed trials … OR = .44
Protective effect of polyglycotic acid was
51% lower in the unclearly concealed trial
(ROR = .49; 95% CI of .35 - .69)
Meta-analysis of Corticosteriods
Meta-analysis of corticosteroids vs. none, after
preterm rupture of membranes, on respiratory
distress syndrome (RDS)
3 adequately concealed trials … OR = 0.72
4 unclearly concealed trials … OR = 0.53
Protective effect of corticosteroids
was exaggerated by 27% in the
unclearly concealed trials
ROR = 0.73; 95% CI of 0.35 - 1.50
Randomization Process (P 138B)
4 OB/GYN
Journals
4 General Medical
Journals
Method of generation of
random numbers
32%
49%
Randomization
concealment
23%
26%
9%
15%
Proper approach to:
Both
In the 4 OB/GYN Journals, found the 2 treatment group sizes to
be much too similar
Expected
Observed
5 ( 5%)
0
48 (50%)
8
Separation of Executer and
Generator
Person(s) who prepare the randomization
scheme should not determine eligibility,
administer treatment, or assess outcome
Neglected critical element
Obviously important—regardless of
methodological quality, access to sequence
and thus opportunity to introduce bias
Faults in this element may be the crack
though which much of the bias seeps into
controlled trials
“Not quite as random as I pretended”
[Martyn C. Not quite as random as I pretended. Lancet 1996; 347:70.]
“Bias in the way treatments are assigned can
be a more powerful determinant of the
outcome of the trial than the treatments that
are being investigated.”
“Trialists … often find it hard to prevent
their hopes that a new treatment might
prove to be beneficial from evolving into
a belief that it really is so.”
RCT in Norway
Examined the effect of Chlorhexidine
compared to placebo on postpartum
infection and Strep B infections in
neonates
Reported as a randomized controlled
trial in the published article
Insert published article
RCT in Norway
Midwives knew on which shelf the
Chlorhexidine and placebo were kept in
the cabinet and just proceeded to
administer whatever they deemed
appropriate
Even if they forgot the shelf, they could
smell the difference
No monitoring of the allocation process
Letter from a Physical Therapist
A “friend of mine” conducted an RCT
comparing 2 training regimens following
surgical repair of the knee for anterior
cruciate ligament damage
“randomization method used was alternating
assignment”
Referred to one patient as “difficult” and
asked the friend to which group she was
assigned
Friend responded “that she was in the group
that required slightly less contact and that her
inclusion was by his choice”
“Not so blind, after all”
Sometimes subverting can work against a
treatment’s apparent effectiveness
“Medical staff … may try to help their sickest
patients by sneaking them into the treatment
group instead of the control group.”
“The drug would then have to be significantly
better than conventional treatment just to appear
equal in efficacy”
[Wallich P. Not so blind after all. Scientific American; May 1996.]
RCTs: Anathema to the
Human Spirit
Scientific need to obtain unbiased data
from an inherently biased source –
human beings
Many trialists are unaware of the
rationale behind RCTs
And the need for trial rigor
Some may intellectually grasp the need,
but once immersed in a trial, have many
contradictory interests (e.g. episiotomy
trial in Canada)
Anathema: A Psychologist’s View
”Fundamental conflict” between scientific
and human imperatives … “Many
probably think they already know what
treatment is best.”
Geoffrey Scobie, psychologist at the
University of Glasgow; Doctors cheat
because they know what’s best
New Scientist, 16 December 95, p. 10
Of course, what we think we ‘know’ is
frequently wrong
Reporting of Exclusions, Double-blinding, and Schedule
Generation Related to the Level of Allocation Concealment
for 250 Trials
Adequately
Concealed
(n=79)
Unclearly
Concealed
(n=150)
Inadequately
Concealed
(n=21)
No Exclusions
(Apparent)
53%
67%
76%
Double-blinding
73%
39%
14%
Adequate generation
of schedule
29%
15%
0%
Authors
Reported