Transcript Public Health Improvement: Evidence base conundrum

Maintaining research rigour in
evaluations of complex interventions
Laurence Moore
Learning Objectives (1)
• To be aware of frameworks for the
development and evaluation of complex
interventions
• To be aware of value (and added value) of
complementary mixed methods
• To understand value of pragmatic CRTs
with embedded process evaluation
• To be aware of examples of successful
CRTs of complex interventions
Learning Objectives (2)
•
•
•
To understand why cluster randomised trials can be the
preferred design
» Situations when individual randomisation is not feasible
» Contamination
» Limitations of Quasi-experimental designs
To be aware of specific issues in the conduct and analysis of
CRTs
» Risk of baseline imbalance
» Intra cluster correlation and the design effect
To be aware of design and analysis strategies to respond to
these issues
» Sample size calculations
» Randomisation methods
» Analysis of cluster randomised trials
Mixed methods
Quantitative
• Methods:
» Outcome / summative
» Intermediate outcomes / process measures
• Research questions:
» What works?
» What is effect?
» How many, how much?
Qualitative
• Methods:
» Observations, field notes, diaries, records,
videos, interviews, focus groups
» Process
• Research questions:
» Why?
» How?
» Barriers / facilitators
Complementary use of mixed
methods: frameworks in health
• Fit for purpose
» Match method to question
• Staged, series of studies
» PRECEDE / PROCEED
» Nutbeam model
» MRC framework
PRECEDE-PROCEED Framework*
Phase 5
Administrative &
policy assessment
Health
Program*
Intervention
Mapping
&
Tailoring*
Phase 4
Educational &
ecological
assessment
Predisposing
Factors
Phase 3
Phase 2
Behavioral & Epidemiological
environmental
assessment
assessment
Phase 1
Social
assessment
Formative evaluation & baselines
for outcome evaluation*
Health
Education
Reinforcing
Factors
Policy
Regulation
Organization
Behavior
Health
Quality of
Life
Environment
Enabling
Factors
Phase 6
Implementation
Phase 7
Process evaluation
Phase 8
Impact evaluation
Phase 9
Outcome evaluation
*New in 4th ed., Green & Kreuter, Health Promotion Planning, in press.
Stages of Research and Evaluation for Health Promotion Programs
Problem
definition
Solution
Generation
Innovation Intervention Intervention
Testing
Demonstration Dissemination
Epidemiology
Intervention
and
demography theory
development
PreSocial,
Testing
behavioural
methods
and
and
organisational
material
Intervention
research
s
literature
search,
Community
metaneeds
analysis
analysis
What is the
problem?
How might it
be solved?
Assessment of outcome
Understanding of
Process
Did the
solution
work?
Can the
program be
repeated/
refined?
Key Research Questions
Program
Monitoring
Assessment
of cost and
benefits
(financial,
social.,
political)
Performance
monitoring
Can the
program be
widely
reproduced?
Can the
program be
sustained?
Phases of RCTs of complex
interventions: MRC April 2000
Complementary use of mixed
methods
• Fit for purpose
» Match method to question
• Staged, series of studies
» PRECEDE / PROCEED
» Nutbeam model
» MRC framework
• In combination within one study
» What works?, how?, for whom? and in what
circumstances?
Public Health Improvement:
Evidence base conundrum
• Good quality trials successfully conducted,
evaluating weak interventions. Small or
zero effect sizes.
• Good quality complex interventions
evaluated using weak research designs.
Biased effect estimates.
Challenges in applying RCTs to
evaluation of complex social interventions
•
•
•
•
•
Recruitment and retention
Scale and Cost
Ethics
Research and Policy Timescales
Implementation
Challenges in applying RCTs to
evaluation of complex social interventions
• Variability in delivery
• Context dependence
• Generalisability / implementation
Types of intervention in which individual
randomisation is difficult or impossible:
• Interventions that entail changing the
organisation of services in a given unit or area
• Interventions targeted at changing the behaviour
of professionals
• Community programmes
• ‘Settings’ based interventions, such as
workplace or school interventions
• Interventions targeted at individuals but based
on social processes
• Risk of contamination
» (though see Puffer & Torgerson)
Problems with quasiexperimental designs
• Selection bias – external & internal validity
• Imbalance at baseline, often not
measurable
• Different trends at baseline
• Ethical considerations
Cluster (group) randomised trials
• ASSIST Peer-led smoking intervention -MRC
» 59 schools randomised
• Fruit tuck shops - FSA
» 43 schools randomised
• Free Breakfast Initiative - WAG
» 111 schools randomised
• Emergency contraception - DH
» 25 schools randomised
Variability in delivery
• RCTs traditionally require that
interventions are standardised and
uniformly delivered
» (efficacy trial)
• Social interventions highly dependent on
quality of delivery
» Value of efficacy trials limited
» eg. school smoking education
• Results of efficacy trials involving enthused
teachers not replicated in roll-out
Efficacy and effectiveness
• Efficacy trial
» To test whether the treatment does more good than
harm when delivered under optimal conditions
• Effectiveness trial
» To test whether the treatment does more good than
harm when delivered via a real-world program in
realistic conditions
» Pragmatic, allowing variability in delivery as would be
experienced in real world
Context dependent
• Social interventions often highly dependent on
the context within which they are delivered
• Argued therefore that RCTs not suited to their
evaluation
• However, RCT design has the advantage that
randomisation process ensures that systematic
differences in external influences between
groups do not occur
» Generally use stratification or minimisation to
minimise imbalance due to small no. of units
» Will achieve unbiased estimate of average effect
Generalisability
• Efficacy trials may demonstrate that
intervention has ‘active ingredients’ that
work
» Effect unlikely to be reproduced in real world
» Attenuated by context and implementation
» Generalisability of small trials with (e.g.) one
educator will be limited
Effectiveness trials with embedded
process evaluation
• Effectiveness trials, implementing interventions
in a manner reproducible in real world
» Realistic level of flexibility allowed, but not adaptation
or reinvention
• Crucial to conduct a comprehensive process
evaluation (largely qualitative) within such a trial
» Monitor variability in context and delivery
» Identify barriers / facilitators
» Relate variability in these factors to variability in
intervention impact
MRC Assist Trial
Peer-led smoking intervention
• Theory based (Diffusion of innovations)
• Developed from similar approach used in
sex education
• Extensively piloted
• Feasibility trial conducted in 6 schools
• Funding for main trial (59 schools) sought
and obtained from MRC
ASSIST Trial
• Intervention led by specialists, as would be the
case if rolled out in the real world
» Not to be implemented by untrained, unmotivated
teachers
• Process evaluation in all 30 intervention schools,
with parallel measures in the 29 control schools
• In-depth process evaluation in sub-sample
• Observations, field notes, diaries, records,
interviews with pupils, teachers, staff
Challenges in embedding process
evaluation within trial
• Hawthorne effects
• Distinguishing team roles
• Differentiating intervention and evaluation
activities
• Volume of data
» Sampling
» Analysis plan
• Power balance
Randomised trials of health promotion
interventions: feasible? valuable?
• Not always!
• Cluster randomised design
• Pragmatic, effectiveness trials
» Unbiased estimate of overall intervention effect
• Additional qualitative and quantitative data
collection to measure variation in context,
process, delivery and outcome
» Identifies issues for further development of
intervention / further testing of its (variable) effect
» Crucial for implementation stage
» Hypothesis generation, not testing
Workshop: Analysis of trials
(cluster randomised)
• Statistical issues
• Design effect, context, implementer,
cluster effects
• Multilevel analysis
• Synthesis of qual/quant data
Cluster randomisation
• Randomise the cluster rather than the individual
• Generally a small number of clusters
» Four per group an absolute minimum
• Use restricted randomisation to ensure balance
in number of clusters per group
• Use stratification or minimisation to minimise
imbalance in group characteristics
• Matched pair design popular, but some
drawbacks
Standard statistical methods, when applied
to cluster randomised trials, will (usually)
lead to:
• Sample size calculations that are too
small
• Confidence intervals that are too narrow
• P-values that are too small
Intra-cluster correlation
• The proportion of the true total variation in
the outcome that can be attributed to
differences between the clusters:


2
b
 
2
b
2
w
Design effect
• The ratio of the variance of the outcome under
the cluster sampling strategy to the variance that
would be expected for a study of the same size
using simple random sampling:
deff = 1 + (n-1)
Sample size inflation
=0.01, m=20
n` = deff * n
n = 360 (or 18 classes) per group
deff = 1+(m-1)
= 1+(20-1).01 = 1.19
n` = 1.19 * 360 = 428.4.
i.e. 429 pupils per group
Number of classes required = 429/20 = 21.4
i.e. 22 classes per group
Sample size inflation
=0.02, m=200
n` = deff * n
n = 360 (or 2 schools) per group
deff = 1+(m-1)
= 1+(200-1).02 = 4.98
n` = 4.98 * 360 = 1792.8.
i.e. 1793 pupils per group
Number of schools required = 1793/200 = 9.0
i.e. 9 schools per group
Further reading
•
•
•
•
•
•
Donner A, Klar N. Design and analysis of cluster randomization trials in
health research. London: Arnold, 2000.
Murray DM. Design and analysis of group-randomized trials. Oxford: OUP,
1998.
Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney GJ, Donner A.
Evaluation of health interventions at area and organization level. BMJ
1999:319:376-379.
Ukoumunne OC, Gulliford MC, Chinn S, Sterne JAC, Burney GJ. Methods
for evaluating area-wide and organisation-based interventions in health and
health care: a systematic review. Health Technol Assess 1999;3(5).
(http://www.hta.nhsweb.nhs.uk/).
Elbourne DR, Campbell MK. Extending the CONSORT statement to cluster
randomised trials: for discussion. Stats Med 2001;20:489-496.
Puffer S, Torgerson D, Watson J. Evidence for risk of bias in cluster
randomised trials: review of recent trials published in three general medical
journals. BMJ, Oct 2003; 327: 785 - 789
www.cardiff.ac.uk/schoolsanddivisions/academicschools/socsi/staff/acad/moore/
•
•
•
•
•
F. Starkey, L. Moore, R. Campbell, M. Sidaway, M. Bloor. Rationale, design
and conduct of a comprehensive evaluation of a school-based peer-led antismoking intervention in the UK: the ASSIST cluster randomised trial
[ISRCTN55572965]. BMC Public Health 2005, 5:43. 22nd April 2005.
http://www.biomedcentral.com/1471-2458/5/43
L. Moore, A. Graham, I. Diamond. On the feasibility of conducting
randomised trials in education: case study of a sex education intervention.
British Education Research Journal 2003;29:673-689.
L. Moore, R. Campbell, A. Whelan, N. Mills, P. Lupton, E. Misslebrook, J.
Frohlich. Self-help smoking cessation in pregnancy: a cluster randomised
controlled trial. British Medical Journal 2002;325:1383-1386.
A. Graham, L. Moore, D. Sharp I. Diamond. Improving teenagers’
knowledge of emergency contraception: results of a cluster randomised
trial. British Medical Journal 2002;324:1179-1183.
L. Moore, C. Paisley, A. Dennehy (2000) Are fruit tuck shops in primary
schools effective in increasing pupils’ fruit consumption? A randomised
controlled trial, Nutrition and Food Science 30(1) 35-38.
Analysis of trials
(cluster randomised)
• Analysis plan
»
»
»
»
Multiple outcomes
Primary and secondary analyses – adjust for baseline / stratifiers
A priori plan (register and publish) – not baseline testing
To include effect modifiers
• Design effect
» Clustering
» Practitioner effect
» Even in individual RCTs across clusters (context effect)
• Multilevel analysis
» Hypothesis generation / informing implementation
• Synthesis
» Triangulation, discordance
• Nothing worse than a poorly conducted
trial
• Complex intervention trials very
challenging
• DO IT WELL – GET ADVICE!!
Cluster randomised trials
Laurence Moore
Cardiff Institute of Society, Health and Ethics
Email: [email protected]
Tel:
02920 875387