Transcript Epidemiology:

Randomized Controlled Trials:
Allocation Concealment
Kenneth F. Schulz, PhD
Fixation on Sequence Generation

Traditionally, many medical researchers
mistakenly consider the sequence
generation process as “randomization”

They frequently ignore allocation
concealment

Without adequate allocation concealment,
proper random sequences can be subverted

For example, a properly generated sequence
posted on a bulletin board
Allocation Concealment

Prevents selection bias by
concealing the allocation sequence
from those assigning participants to
intervention groups

Until the moment of assignment
Mechanisms for Selection Bias
in RCTs


If those responsible for entering
participants know, or can detect, the
upcoming treatment allocations:

Decide entrance based on that knowledge

Channel participants with a better prognosis
to the experimental group and those with a
poorer prognosis to the control group, or vice
versa
In any case, they introduce bias into the
treatment comparison
Do Not Confuse Allocation
Concealment with Blinding

Allocation concealment seeks to prevent
selection bias, protects assignment
sequence before and until allocation, and
can always be successfully implemented

In contrast, blinding seeks to prevent
ascertainment bias, protects sequence
after allocation, and cannot always be
successfully implemented
Importance of Allocation
Concealment

Unclearly concealed and inadequately
concealed trials, compared to
adequately concealed trials,
exaggerated the estimates of an
intervention’s effectiveness by 30% to
40%, on average
Schulz KF, Chalmers I, Hayes RJ, Altman DG. Empirical
evidence of bias: dimensions of methodological quality
associated with estimates of treatment effects in controlled
trials. JAMA 1995;273:408-412.
Similar Results on Allocation
Concealment in a Separate Study

Replicated our methods in different
subject areas (digestive & circulatory
diseases and mental health)

Examined 11 meta-analyses

Inadequately concealed trials exaggerated
estimates of effectiveness by 37%
[Moher D, et al. Does quality of reports of randomised trials
affect estimates of intervention efficacy reported in metaanalyses? Lancet 1998; 352: 609-13.]
Allocation Concealment: Summary



Adequate allocation concealment emerges
from our analyses as crucial to reducing
bias
Without it, the whole point of
randomisation vanishes and bias may
distort results
Our results support Mosteller and his
colleagues
“When the randomization leaks, the trial’s
guarantee of lack of bias runs down the drain”
Method of allocation of treatment in 208
controlled trials in head injury
Dickinson K, et al., BMJ 2000;320:1308-1311.
Method of allocation
No. of trials
Adequate
Centralised randomisation by telephone
Numbered/coded identical containers administered sequentially
1
11
Randomisation scheme controlled by pharmacy
8
Sequentially numbered, sealed, opaque envelopes
2
Not adequate
Other
18
Date of birth
1
Day of week
3
Alternation
3
Not stated
161
Actual allocation method with the
method just specified as “random”

Difficult (Peto and I. Chalmers experience)

MA of OCs

David G. knew the PI who put him in
contact with the statistician

Described a “great” method . . . Post-it
notes!


Seemed indignant . . . We asked for clarification
Perfect also for stratification—different colors!!
Personal Accounts of Deciphering
Assignment Sequences

Conducted over 20 epidemiological
workshops for medical residents and
medical junior faculty

Each included 20-25 participants

Asked how many of the participants had
deciphered, or had witnessed someone
else decipher, an assignment sequence
Personal Accounts of Deciphering
Assignment Sequences (cont.)

With assurance of anonymity, more
than half related at least one
instance of deciphering

Not half of all trials


No accurate denominator
Nevertheless, not a rare occurrence
Decipherings Run the Gamut
From Simple to Intricate

Simple the most frequent and usually
take advantage of inadequate
allocation concealment schemes:




Posting sequence on a bulletin board
Opening unsealed envelopes
Translucent envelopes to a light bulb
Opening many envelopes that were not
sequentially numbered
Intricate Decipherings Less Frequent

Needed to circumvent more adequate
allocation schemes

With sequentially numbered drug
containers




Based on appearance of tablets in
unsealed containers
Appearance of the label
With central allocation, obtaining the
next few allocations at once
Envelopes to a “hot light” in radiology
Attempted to Decipher a Numbered
Container Scheme

Gave up

Found attending physician rifling
the P.I.’s files for the assignment
sequence

Horrified? No, impressed with his
brilliance and proceeded to help

Should be kept in a locked location
Ignorance Is Not Bliss
“At the annual meeting of the Society for
Clinical Trials, held in Houston, … Schulz
reported data showing that the experts’ fears
about ignorance are justified. Schulz
examined 250 reports of clinical trials and
came up with an intriguing finding…”
[Rachel Nowak. Problems in clinical trials go far beyond misconduct.
Science 1994; 264: 1539.]
Ignorance Is Not Bliss
“Most physicians are not trained in
basic scientific principles, let alone
clinical trials”
Stephen George, Duke University, Chair of the Statistics
Committee for NCI, NIH Cancer Clinical Cooperative Groups
“I fantasize that in the future (training)
will become a requirement for running
clinical trials”
John Gallin, Director of NIH’s Clinical Center
Rigorous Trials Annoy Humans


Investigators

Certain Ps to benefit

May want the results of study to reveal
the “truth”
Proper trial procedures attempt to
impede human inclinations

Hence, aspects of properly conducted
RCTs annoy investigators
Without Methodological Rigour

The challenge of deciphering may frequently
become too great a temptation to resist

Even without intent to bias

“The only way to get rid of a temptation is to
yield to it.” Oscar Wilde

Deciphering may just reflect human
inquisitiveness and ingenuity

Does not necessarily indicate scientific
malevolence
Without Methodological Rigour

Or deciphering may reflect deliberate
acts to alter findings

Whatever the motivations, innocent
or naïve or deliberate, those actions
undermine the validity of the trial

Investigators must devote diligent,
persistent attention to randomization
Minimal Standards: Common
Allocation Concealment Approaches

Sequentially numbered, opaque,
sealed envelopes (SNOSE)

Pharmacy control

Numbered or coded containers

Central randomization

Realistically, these standards
should be exceeded
Envelopes for Allocation
Concealment

More susceptible to manipulation
through human ingenuity

Less than ideal method of allocation
concealment

If used, investigators must diligently
develop and monitor the process
Envelopes (Cont.)

SNOSE

Ensure that the envelopes are opened
sequentially

only after the P’s name
and other details are
written on the appropriate
envelope
Envelopes (Cont’d)

Pressure-sensitive or carbon paper
inside





Supports enrolment
Encourages orderly opening
Promotes proper assignment
Creates valuable audit trail
Cardboard or aluminum foil inside
envelope
“Pharmacy Controlled”

Compliance with randomization/allocation
concealment methods questionable

Aware of gross distortions
 Pharmacy ran out of one drug . . .
 Alternate assignment

Should not assume pharmacists
knowledge

Ensure they, and all research partners,
follow proper trial procedures
“Numbered Containers”

Excellent – prevents foreknowledge

Must take proper precautions

Appearance – exactly the same

Weight – exactly the same

Sound when shaken – exactly the same

Lettering – exactly the same

Odor – exactly the same

Sealed

Difference between concealment and blinding
“Central Randomization”

Frequently telephone







Fax
Email
Internet
“Computer”
Assignments only provided after
participants irrevocably enrolled
Must check procedures for leaks
Monitor process for adherence
RCTs: Anathema to the Human Spirit

Must acknowledge the vagaries of
human nature

Must establish methodological
safeguards that thwart attempts to
contaminate trials with bias
Subversions of Sequences

Conscious or subconscious?

Direct or compensatory?

Doesn’t matter
 Biases the trial
Participants
Randomize
Placebo
40% Noncompliance
60%
Compliance
New Oral Drug
25% Noncompliance
75%
Compliance
Group representing the
policy of no treatment
Group representing the
policy of oral treatment
Outcome
Outcome
RCT Compared the Effectiveness of Clofibrate in
Preventing Cardiac Deaths in Men Who Had
Survived a Myocardial Infarction
Clofibrate
Placebo
5 Year mortality
20.2%
20.9%
(p = .55)
Eliminating deviates from
clofibrate
(80% adherence)
15.0%
20.9%
(p < .05)
Eliminating deviates from
both groups
15.0%
15.1%
Authors state that:
• One can justify almost any conclusion, dependent upon the analysis
chosen
• Manipulating deviates leads to severe bias
• Can you ever do so?
Exclusions After Randomization

Can introduce bias and should be
carefully scrutinized

All randomized patients should be
analyzed, and analyzed as part of the
group to which they were initially
assigned

ITT (Intention-to-treat)
Exclusions of LFU Damage
Internal Validity

Without outcomes from those lost to
follow-up, investigators have little
choice but to exclude them from the
analysis

Any losses damage internal validity


However, differential rates of loss among
comparison groups cause major damage
Investigators must minimize their
losses to follow-up
Retention of Trial Participants


Minimizing losses exudes difficulties
Eliminating losses may be impossible


But investigators too frequently profess
insurmountable difficulties
Most should work harder to obtain
higher follow-up rates

Investigators must commit adequate
attention and resources to develop and
implement procedures to minimize
losses
Innovative Twists that Cultivate
High Follow-up Rates

Establish many conveniently placed FU
facilities


Too often investigators expect participants to
visit a single, inconvenient location
Shortening data collection form caters to the
participants’ wishes




Long sessions lead them to vote with their feet
Be considerate
Foster follow-up by not overburdening
Also may engender higher quality data on the
main items
END
Meta-analysis of Polyglycolic
Meta-analysis of polyglycolic acid versus
catgut for perineal repair on short term pain
1 adequately concealed trial … OR = .89
4 unclearly concealed trials … OR = .44
Protective effect of polyglycotic acid was
51% lower in the unclearly concealed trial
(ROR = .49; 95% CI of .35 - .69)
Meta-analysis of Corticosteriods
Meta-analysis of corticosteroids vs. none, after
preterm rupture of membranes, on respiratory
distress syndrome (RDS)
3 adequately concealed trials … OR = 0.72
4 unclearly concealed trials … OR = 0.53
Protective effect of corticosteroids
was exaggerated by 27% in the
unclearly concealed trials
ROR = 0.73; 95% CI of 0.35 - 1.50
Randomization Process (P 138B)
4 OB/GYN
Journals
4 General Medical
Journals
Method of generation of
random numbers
32%
49%
Randomization
concealment
23%
26%
9%
15%
Proper approach to:
Both
In the 4 OB/GYN Journals, found the 2 treatment group sizes to
be much too similar
Expected
Observed
5 ( 5%)
0
48 (50%)
8
Separation of Executer and
Generator

Person(s) who prepare the randomization
scheme should not determine eligibility,
administer treatment, or assess outcome

Neglected critical element

Obviously important—regardless of
methodological quality, access to sequence
and thus opportunity to introduce bias

Faults in this element may be the crack
though which much of the bias seeps into
controlled trials
“Not quite as random as I pretended”
[Martyn C. Not quite as random as I pretended. Lancet 1996; 347:70.]
“Bias in the way treatments are assigned can
be a more powerful determinant of the
outcome of the trial than the treatments that
are being investigated.”
“Trialists … often find it hard to prevent
their hopes that a new treatment might
prove to be beneficial from evolving into
a belief that it really is so.”
RCT in Norway

Examined the effect of Chlorhexidine
compared to placebo on postpartum
infection and Strep B infections in
neonates

Reported as a randomized controlled
trial in the published article
Insert published article
RCT in Norway

Midwives knew on which shelf the
Chlorhexidine and placebo were kept in
the cabinet and just proceeded to
administer whatever they deemed
appropriate

Even if they forgot the shelf, they could
smell the difference

No monitoring of the allocation process
Letter from a Physical Therapist

A “friend of mine” conducted an RCT
comparing 2 training regimens following
surgical repair of the knee for anterior
cruciate ligament damage


“randomization method used was alternating
assignment”
Referred to one patient as “difficult” and
asked the friend to which group she was
assigned

Friend responded “that she was in the group
that required slightly less contact and that her
inclusion was by his choice”
“Not so blind, after all”
Sometimes subverting can work against a
treatment’s apparent effectiveness
“Medical staff … may try to help their sickest
patients by sneaking them into the treatment
group instead of the control group.”
“The drug would then have to be significantly
better than conventional treatment just to appear
equal in efficacy”
[Wallich P. Not so blind after all. Scientific American; May 1996.]
RCTs: Anathema to the
Human Spirit

Scientific need to obtain unbiased data
from an inherently biased source –
human beings

Many trialists are unaware of the
rationale behind RCTs


And the need for trial rigor
Some may intellectually grasp the need,
but once immersed in a trial, have many
contradictory interests (e.g. episiotomy
trial in Canada)
Anathema: A Psychologist’s View

”Fundamental conflict” between scientific
and human imperatives … “Many
probably think they already know what
treatment is best.”

Geoffrey Scobie, psychologist at the
University of Glasgow; Doctors cheat
because they know what’s best
New Scientist, 16 December 95, p. 10

Of course, what we think we ‘know’ is
frequently wrong
Reporting of Exclusions, Double-blinding, and Schedule
Generation Related to the Level of Allocation Concealment
for 250 Trials
Adequately
Concealed
(n=79)
Unclearly
Concealed
(n=150)
Inadequately
Concealed
(n=21)
No Exclusions
(Apparent)
53%
67%
76%
Double-blinding
73%
39%
14%
Adequate generation
of schedule
29%
15%
0%
Authors
Reported