Transcript Slide 1

DISABILITY RESEARCH TO PRACTICE
SESSION 5:
Conducting Systematic Reviews of Evidence-Based Disability and Rehabilitation Research
Dr Carole Torgerson
Reader
Institute for Effective Education, University of York
January 20, 2010
(3:00 PM Eastern)
National Center for the Dissemination of Disability Research (NCDDR)
Funded by NIDRR, US Department of Education, PR# H133A060028
© 2010 by SEDL
Why do we need to quality
appraise trials?
• In reading individual trials:
»
to differentiate between and within trials regarding quality issues;
• In a systematic review:
» To investigate whether the individual studies in the review are affected
by bias. Systematic errors in a study can bias its results by
overestimating or underestimating effects. Bias in an individual study or
individual studies in the review can in turn bias the results of the review
» To make a judgement about the weight of evidence that should be
placed on each study (higher weight given to studies of higher quality in
a meta-analysis)
» To investigate differences in effect sizes between high an low quality
studies in a meta-regression
• In designing trials:
» To use items of quality issues to ensure that the trial is methodologically
rigorous.
2
“A careful look at randomized
experiments will make clear that
they are not the gold standard.
But then, nothing is. And the
alternatives are usually worse.”
Berk RA. (2005). Journal of Experimental Criminology, 1, 417-433.
3
Characteristics of a rigorous trial
• Once randomised, all participants are
included within their allocated groups.
• Random allocation is undertaken by an
independent third party.
• Outcome data are collected blindly.
• Sample size is sufficient to exclude an
important difference.
• A single analysis is pre-specified before
data analysis.
4
Education: Comparison with
Health Education
Torgerson CJ, Torgerson DJ, Birks YF, Porthouse J. (2005) A comparison of
randomised controlled trials in health and education. British Educational
Research Journal, 31: 761-785. (based on n = 168 trials)
5
Problems with RCTs
• Failure to keep to random allocation
• Attrition can introduce selection bias
• Unblinded ascertainment can lead to
ascertainment bias
• Small samples can lead to Type II error
• Multiple statistical tests can give Type I
errors
• Poor reporting of uncertainty (e.g., lack of
confidence intervals).
6
Independent assignment
• “Randomisation by centre was conducted
by personnel who were not otherwise
involved in the research project” [1]
• Distant assignment was used to: “protect
overrides of group assignment by the staff,
who might have a concern that some
cases receive home visits regardless of the
outcome of the assignment process” [2]
[1] Cohen et al. (2005). J of Speech Language and Hearing Research, 48, 715-729.
[2] Davis RG, Taylor BG. (1997). Criminology, 35, 307-333.
7
Attrition
• Attrition can lead to bias; a high quality trial will
have maximal follow-up after allocation.
• It can be difficult to ascertain the amount of
attrition and whether or not attrition rates are
comparable between groups.
• A good trial reports low attrition with no between
group differences.
• Rule of thumb: 0-5%, not likely to be a problem.
6% to 20%, worrying, > 20% selection bias.
8
Concealed allocation –
why is it important?
» Good evidence from multiple sources shows that
effect sizes for RCTs where randomisation was not
independently conducted were larger compared with
RCTs that used independent assignment methods.
» A wealth of evidence is available that indicates that
unless random assignment was undertaken by an
independent third party, then subversion of the
allocation may have occurred (leading to selection
bias and exaggeration of any differences between the
groups).
9
Allocation concealment:
a meta-analysis
• Schulz and colleagues took a database of 250
randomised trials in the field of pregnancy and
child birth.
• The trials were divided into 3 groups with
respect to concealment:
» Good concealment (difficult to subvert);
» Unknown (not enough detail in paper);
» Poor (e.g., randomisation list on a public notice
board).
• They found exaggerated effect sizes for poorly
concealed compared with well concealed
randomisation.
10
Comparison of adequate, unclear
and inadequate concealment
Allocation
Concealment
Adequate
Effect Size
OR
1.0
Unclear
0.67
Inadequate
0.59
P < 0.01
Schulz et al. JAMA 1995; 273:408.
11
Summary of assignment and
concealment
• Judge whether RCT or quasi-experiment or other.
• Increasing evidence to suggest that subversion of
random allocation is a problem in randomised trials. The
‘gold-standard’ method of ‘random’ allocation is the use
of a secure third party method.
• Judge whether or not the trial reports that an
independent method of allocation was used. Poor
quality trials: use sealed envelopes; do not specify
allocation method; or use allocation methods within the
control of the researcher (e.g., tossing a coin).
• Judge whether there are assignment discrepancies, e.g.
failure to keep to random allocation
12
Other design issues
• Unblinded ascertainment (outcome
measurement) can lead to ascertainment bias
• Small samples can lead to Type II error
(concluding there is no difference when there is
a difference)
• Attrition (drop-out) can introduce selection bias
• Multiple statistical tests can give Type I errors
(concluding there is a difference when this is
due to chance)
• Poor reporting of uncertainty (e.g., lack of
confidence intervals).
13
Blinding of participants
and investigators
• Participants can be blinded to:
» Research hypotheses
» Nature of the control or experimental condition
» Whether or not they are taking part in a trial
• This may help to reduce bias from resentful
demoralisation
• Investigators should be blinded (if possible) to
follow-up tests as this eliminates ‘ascertainment’
bias. This is where consciously or
unconsciously investigators ascribe a better
outcome than is the truth based on the
knowledge of the assigned groups.
14
Blinding of outcome assessment
• Judge whether or not post-tests were administered by
someone who is unaware of the group allocation.
Ascertainment bias can result when the assessor is not
blind to group assignment, e.g., homeopathy study of
histamine showed an effect when researchers were not
blind to the assignment but no effect when they were.
• Example of outcome assessment blinding: Study “was
implemented with blind assessment of outcome by
qualified speech language pathologists who were not
otherwise involved in the project”
Cohen et al. (2005) J of Speech Language and Hearing Research 48, 715-729.
15
Intention to Treat (ITT)
• Randomisation ensures the abolition of selection
bias at baseline; after randomisation some
participants may cross over into the opposite
treatment group (e.g., fail to take allocated
treatment or obtain experimental intervention
elsewhere).
• There is often a temptation by trialists to analyse
the groups as treated rather than as randomised.
• This is incorrect and can introduce selection bias.
16
ITT analysis: examples
• Seven participants allocated to the control
condition (1.6%) received the intervention,
whilst 65 allocated to the intervention failed to
receive treatment (15%). The authors,
however, analysed by randomised group CORRECT approach.
Davis RG, Taylor BG. (1997) Criminology, 35, 307-333.
17
ITT analysis: examples
• “It was found in each sample that approximately
86% of the students with access to reading
supports used them. Therefore, one-way
ANOVAs were computed for each school sample,
comparing this subsample with subjects who did
not have access to reading supports.”
- INCORRECT
Feldman SC, Fish MC. (1991) Journal of Educational Computing Research 7, 25-36.
18
Statistical power
• Few effective educational interventions produce
large effect sizes especially when the
comparator group is an ‘active’ intervention. In a
tightly controlled setting 0.5 of a standard
deviation difference at post-test is good. Smaller
effect sizes in field trials are to be expected (e.g.
0.25). To detect 0.5 of an effect size with 80%
power (sig = 0.05), we need 128 participants for
an individually randomised experiment.
19
Survey of trial quality
Characteristic
Cluster Randomised
Sample size justified
Concealed randomisation
Blinded Follow-up
Use of CIs
Low Statistical Power
Drug Health Education
1%
36%
18%
59% 28%
0%
40%
8%
0%
53% 30%
14%
68% 41%
1%
45% 41%
85%
Torgerson CJ, Torgerson DJ, Birks YF, Porthouse J. (2005). A comparison of
randomised controlled trials in health and education. British Educational
Research Journal, 31:761-785. (based on n = 168 trials)
20
CONSORT
• Because the majority of health care trials
were badly reported, a group of health
care trial methodologists developed the
CONSORT statement, which indicates key
methodological items that must be
reported in a trial report.
• This has now been adopted by all major
medical journals and some psychology
journals. www.consort-statement.org
21
The CONSORT guidelines, adapted
for trials in educational research
•
•
•
•
•
•
•
•
Was the target sample size adequately determined?
Was intention to teach analysis used? (i.e. were all children who
were randomised included in the follow-up and analysis?)
Were the participants allocated using random number tables, coin
flip, computer generation?
Was the randomisation process concealed from the investigators?
(i.e. were the researchers who were recruiting children to the trial
blind to the child’s allocation until after that child had been included
in the trial?)
Were follow-up measures administered blind? (i.e. were the
researchers who administered the outcome measures blind to
treatment allocation?)
Was precision of effect size estimated (confidence intervals)?
Were summary data presented in sufficient detail to permit
alternative analyses or replication?
Was the discussion of the study findings consistent with the data?
22
NPT Checklist
• Includes a ten item checklist
• Includes issues relating to:
» The standardisation of the intervention
» Care provider influences
» Measures to minimise bias from a lack of blinding
Boutron et al., 2005. A checklist to evaluate a report of a nonpharmacological trial
(CLEAR NPT) was developed using consensus. Journal of Clinical Epidemiology,
58(12), 1233-12240.
23
Flow Diagram
• In health care trials reported in the main
medical journals authors are required to
produce a CONSORT flow diagram.
24
Year 7 Pupils
N = 155
Randomised
ICT group
N = 77
N = 3 left school
No ICT Group
N = 78
N = 1 left school
70 valid pre-tests
67 valid post-tests
63 valid pre and post
75 valid pre-tests
71 valid post-tests
67 valid pre and post tests
25