Transcript Combining different sources of evidence

Synthesis of multiple
forms of evidence
Professor Elizabeth Waters
Dr Belinda Hall
The McCaughey Centre
VicHealth Centre for the Promotion of Mental Health and Community Wellbeing
Melbourne School of Population Health
Synthesis…
...refers to a combination of two
or more entities that together
form something new
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Complex systems
• Systems Thinking
– An overall approach to problem solving:
component parts of system best
understood in context of relationships with
each other and with other systems
– View “problems” as parts of an overall
system, rather than reacting to specific
parts/outcomes
– Examine the linkages and interactions
between elements
– Cyclical thinking rather than linear cause
and effect
• Need multiple forms of evidence to
understand the system
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
MRC framework for complex
interventions
1 Key elements
the development and evaluation
KeyFigure
elements
of the of
development
andprocess
evaluation proce
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Reviews of complex
interventions
• Evaluating complex interventions poses a
challenge and requires a substantial
investment of time
• Unless the intervention studies illuminate
processes and mechanisms they often fail
to provide useful information
• If the result is negative, is it because:
• the intervention is inherently ineffective (either
because the intervention was inadequately
developed or because all similar interventions
are ineffective)?
• it was inadequately applied or applied in an
appropriate context?
• the trial used an inappropriate design,
comparison group or outcomes?
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Heterogeneity: the antithesis of
synthesis?
• Multiple forms of evidence will produce
heterogeneity
• How much are we comfortable with?
• Different forms of difference:
– Evidence that answers different questions:
feature in different parts of the review (eg.
theory vs effectiveness data)
– Answering a similar question in a
different way (eg. different study
designs, populations, interventions,
outcomes)
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Narrative synthesis
Framework1:
• Developing a theoretical model of how
the intervention works, why and for
whom
• Developing a preliminary synthesis of
findings of included studies
• Exploring relationships in the data
• Assessing the robustness of the
synthesis
1Popay
et al. 2006. Guidance on the conduct of narrative synthesis in systematic reviews.
ESRC Methods Programme.
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Example: Health and social
effects of housing improvements
(Thomson et al, in press)
• Synthesis structure:
– Type of housing intervention
• Context and population
– Outcomes (respiratory health, general health,
mental health)
» Study quality
» Study design
• Data represented in several ways
– Forest plot for similar quantitative
outcomes without pooling (due to
heterogeneity)
– Table visually representing the direction of
health impact from all included studies
– Logic model using review data to show
nature and direction of health and socioeconomic impacts following housing
improvements
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Logic model developed using
review data (Thomson et al, in
press)
Logic model mapping impact types and direction, and links to health impacts
reported in qualitative and quantitative studies of modern day housing
improvements in developed world (warmth & energy efficiency improvements,
and rehousing/retrofitting)
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Using qualitative synthesis
to explore heterogeneity
2 reviews:
• Qualitative review on HIV/AIDS patients perspectives on adherence
• Cochrane Review of intervention trials to improve adherence to therapy in
HIV/AIDS patients
Tabulated whether the interventions in the CR corresponded with the patients’
perspectives on how to help them
Candy et al. BMC Medical Research Methodology 2011, 11:124
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Take home messages
• Synthesis must generate something
“new”
• A range of evidence forms should be
considered in developing and
evaluating complex interventions
• Frameworks are useful for organising
the different forms and aligning these
with the questions they answer
• Commit to including process and
contextual information as standard
‘evidence synthesis’ practice
• Heterogeneity is a way of life
– Contextual information helps us make
sense of it
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health
Acknowledgements
• Hilary Thomson and the housing
review team
• CPHG Review authors
• CPHG Editors
• Sasha Shepperd
The McCaughey Centre VicHealth Centre for the Promotion of Mental Health
and Community Wellbeing
Melbourne School of Population Health