Transcript No Slide Title

Methods Symposium on Complex interventions:
19th Cochrane Colloquium, Madrid 2011
Setting the co-ordinates: methodological
issues and Cochrane reviews of complex
interventions
Mark Petticrew
Faculty of Public Health and Policy
London School of Hygiene & Tropical Medicine
The inverse evidence law
• The best evidence we have is about
the most simple interventions. We
have less, or weaker evidence about
complex interventions – such as
policies
• But policymakers are often most
interested in the complex questions:
• “Many policy choices cannot be broken
down into discrete interventions
evaluated in a context-free manner.”
(Kouri, 2009)
What are complex interventions?
• “Complex interventions are
…interventions that are not drugs or
surgical procedures, but have many
potential “active ingredients... A complex
intervention combines different
components in a whole that is more than
the sum of its parts.” (Oakley et al. BMJ
2006)
Other aspects of complex interventions
to consider …
• Theory underlying the intervention
• Mechanisms and pathways by which the
intervention affects (or may affect) outcomes
• Lack of linear, well-evidenced causal pathways
linking the intervention and the health
outcomes
• Mediators (causal mechanisms) and
moderators (characteristics of populations,
settings etc) of outcomes
• Aspects of context and setting and how they
interact with the intervention
• Feedback loops, synergies, phase effects
Implications of complexity for…
Defining the research
question
Implications for reviews of complex
interventions
• Getting the review question right is even more
important than usual.
• Population: (Individuals, families, communities?)
• Intervention: What is the intervention? Delivered at
what level? Do we lump or split? (Broad vs narrow
questions; components vs whole systems)
• Comparison: Are controls possible? Are RCTs
possible? (RCTs of area-based interventions, or
changes in organisations/systems are few and far
between…)
• Outcomes: What outcomes? At what levels?
Include health and non-health outcomes?
Going beyond PICO
• Context: How do we identify context? How do
we assess how it interacts with the
intervention?
• Processes: Where do we find, and how do we
integrate this information?
• Where does theory fit in?: Helps describe the
causal pathways linking the intervention
through intermediate outcomes to the final
health and other outcomes; helps understand
the purpose and assumptions underlying
“components” of interventions and how they
interact; informs evidence synthesis (e.g.
Hawe et al., 2004)
Evaluations of complex social
interventions can involve
•
Randomised Controlled Trials;
•
Quasi-experimental study designs, using
control/comparison groups/areas where
appropriate
•
Uncontrolled studies (e.g. time series analysis,
before-and-after studies) – what we know about the
effects of smoking bans comes from ITS and
uncontrolled B/A studies
•
“Case studies”; qualitative research; other types of
data
•
Even (gasp!)… Non-scientific information
•
Depending on the research question
Some things it might be helpful to know
about (when looking for evidence of the
effects of complex interventions)
• Mechanisms and pathways by which the
intervention brings about the desired
intermediate and final outcomes (including
unintended adverse effects)
• Mediators (causal mechanisms) and
moderators (characteristics of studies,
populations, settings etc)
• Aspects of context and setting
• Feedback loops, synergies, phase effects
(Maybe)
• Theory underlying the intervention
• …Depending on the research question
Where do we find all this?
• It may be helpful to think of the
sources of complexity (rather than
complexity itself), and map each of
these onto specific sources of
evidence
Mapping aspects of complexity to
types of evidence and study designs
• Source of complexity: Multiple
components; interactions between
components
• Evidence needed: of the independent
effects of components of the intervention,
and interactions between those
components.
• This may derive from quantitative or
qualitative data. Studies with factorial
designs may also explore these effects.
Individual studies with different
configurations of components may be
included in a review allowing indirect
comparisons between studies. Metaregression may also be of value.
Feedback loops
• Source of complexity: Feedback loops
• Evidence needed: Evidence of feedback
loops may derive from qualitative studies
carried out alongside trials; or may be
described in qualitative research.
Longitudinal studies carried out as part of
process evaluations may be of value.
Phase transitions
• Source of complexity: Phase
transitions
• Source of evidence: Some longitudinal
element is necessary to identify these. For
example changes in direction or size of
effects over time may be observed in
studies with multiple data points (e.g., ITS
studies).
• Qualitative data may also be available to
describe phase changes.
• Source of complexity: Multiple (health
and non-health) outcomes
• Source of evidence: Any type of
evaluative study - qualitative studies
may also show the range direction
of effects. The Cochrane Handbook:
“it is important to include all
outcomes that are likely to be
important to users) but overall
conclusions are more difficult to
draw if there are multiple
analyses)”.
• Important to state in the protocol
which analyses and outcomes are of
interest. Outcomes should be
classified in advance as primary and
secondary outcomes
• Source of complexity: Effects at
different levels
• Source of evidence: Cluster RCTs may
provide outcome data at both cluster, and
individual level; studies may collect data
from individuals about effects not just on
themselves, but on their families,
communities etc.
• External data sources (e.g. routine data)
may show effects at these levels.
Qualitative studies have also been used to
explore effects of interventions at multiple
levels.
Source of complexity
Study design or source of data for this type of complexity will include…
Multiple components
Synergies/interactions
(between
components of an
intervention)
Evidence of the independent effects of components of the intervention, and
interactions between those components. This evidence may be available in
the form of either quantitative or qualitative data. Studies with factorial
designs may also explore these effects. Individual studies with different
configurations of components may be included in a review allowing indirect
comparisons between studies.18 Meta-regression may also be of value.
Flexibility/tailoring/nonstandardisation
of implementation
This information may come from process evaluations; studies describing
implementation; policy documents and other sources.3
Feedback loops
Evidence of feedback loops may derive from qualitative studies carried out
alongside trials; or may be described in qualitative or quantitative research
(e.g., structural equation modelling). Longitudinal studies carried out as
part of process evaluations may be of value.
Phase transitions
Some longitudinal element is necessary to identify these. For example changes
in direction or size of effects over time may be observed in studies with
multiple data points (e.g., ITS studies). Qualitative data may also be
available to describe phase changes.
Multiple outcomes.6
Data on multiple health and non-health outcomes may come from any type of
evaluative study - qualitative studies may also show the range and nature
(direction) of effects.
Effects at different
levels
Evidence may come from may come from any type of evaluative study; e.g.
cluster RCTs may provide outcome data at both cluster, and individual
level; studies may collect data from individuals about effects not just on
themselves, but on their families, communities etc. External data sources
(e.g. routine data) may show effects at these levels. Qualitative studies
have also been used to explore effects of interventions at multiple levels,
e.g. housing interventions.19
Moderating effects of
context
Requires a clear a priori definition of what aspects of context are of interest to the
review. Evidence of contextual effects may emerge from quantitative
analyses as part of the original study (e.g. subgroup and sensitivity
analyses). It may also be described in qualitative research carried out as
one component of a trial. Policy documents may describe the policy
context.3 Mediators and moderators may be explored in meta-analysis.20
Finally…a sceptical
view of complexity...
• “ Complexity is a strategy used by
professional elites to maintain
control. Proclaiming that a
problem is complex is shorthand
for saying that you have no role in
solving it.” – Ian Roberts, Phil
Edwards “The Energy Glut” 2011
• Just because the problem is
complex, doesn’t always mean
that an equally complex
explanation or analysis is always
helpful to users
• Answering simpler questions is
good, too, and is often the
essential first step
CHEMG
CPHRG
CDC
EPOC
…