Transcript Data Analysis and Synthesis March 2011

Mixed method reviews
Data analysis and synthesis
Complexity
“Some problems are so
complex that you
have to be highly
intelligent just to be
undecided about
them”
Lawrence J. Peter
Narrative synthesis (NS)


An approach to the synthesis of findings from
multiple studies that relies primarily on the
use of words and text to summarise and
explain the findings of the synthesis
Designed for integrating effectiveness and
implementation data, but can be used to
synthesise evidence for a wide range of
questions including effectiveness,
experience, need, process, implementation
etc.
Using a synthesis method
Advantages
 Assists in review team communication and
documentation
 Assists in making the processes of synthesis
available to others for validation
 Assists with the presentation and
communication of the results of synthesis
Guidance

Pope C, Mays N, Popay J (2007) Synthesising
Quantitative and Qualitative Evidence: a guide to
methods. Milton Keynes, Open University Press
Popay J et al. (2006) Guidance on the Conduct of
Narrative Synthesis in Systematic Reviews.
Available at:
http://www.lancs.ac.uk/shm/research/nssr/research/dis
semination/publications/NS_Synthesis_Guidance_v
1.pdf

What NS is, and isn’t



A framework to
increase the
transparency and
trustworthiness of the
synthesis
Includes description of
tools and techniques
Could be used to
synthesise the findings
from mixed evidence
sources



A guide to the whole
process of systematic
review – only the stage
of synthesis
A cookbook of tools
and techniques to be
selected at random
Aims to be accessible,
but not a guide for
complete novices in
research
Relationship to systematic
review
Iterative not linear
Question development
Theoretical framework
Search/data extraction
Quality assessment
Descriptive analysis
Preliminary synthesis
Interpretive analysis
Final modelling/synthesis
Theory testing
/development
Looking for
patterns
in data
Generalise on basis of theory
Phases of NS





Phase 1: Develop a theoretical framework*
Phase 2: Descriptive analysis + develop a
preliminary synthesis
Phase 3: Interpretive analysis: explore
relationships in the data + construct synthesis
Phase 4: Assess the robustness of the
synthesis
NB Not linear stages: phases are iterative
*not always required
Theoretical framework


Purpose: to inform decisions about the review
question, contribute to interpretation of
findings, and to assess the applicability of the
findings
Sources:




Research question
Topic
Theory
Hypotheses
Descriptive analysis



Purpose: to organise findings to provide an initial
description of patterns across the included studies
Tasks: select, group, juxtapose
Tools:







Textual descriptions
Groupings and clusters
Tabulation
Transform to a common rubric
Vote count
Thematic analysis
Content analysis
Descriptive analysis
Descriptive analysis
SELECT
GROUP
JUXTAPOSE
Descriptive analysis
Textual descriptions
Groupings and clusters
Tabulation
Transform to a common rubric
Counting
Thematic analysis
Content analysis
Tools for describing: examples






Textual descriptions: a short ‘story’ with key
messages, significant information that acts as a
‘viewing platform’.
Ordering/grouping: finding shared features and
clustering studies along those lines
Tabulation: Tables summarising key features of study
objectives, design, context, findings
Common rubric: Using a common framework to
allocate material from multiple studies
Counting: Using occurrence or vote counting to
consider the frequency of phenomena in the data
Content/thematic analysis: qualitative analysis
EXERCISE: description



Select summary information from the
narrative description to put in a table, making
sure that the table will fit data from all three
studies
Look at the types of clients, and context. Can
you spot potential groupings?
Look at the findings across the three studies.
Can you spot potential categories or themes?
Output: preliminary synthesis


Descriptive narrative and tabular summary of the
studies and findings
Specification of analytical framework/common rubric






Events, participants, contexts?
Stages, phases, time?
Ordering, grouping?
Positives, negatives?
Hierarchies, key issues?
Preliminary synthesis

Testing the theory with a subset of data to make sure it
works
Exploring relationships in the data

Purpose: to consider relationships between:




study results and key aspects of those studies (within)
factors across included studies (between)
Tasks: compare, combine, translate, integrate
Tools:





Identifying mediators and moderators
Sub-set analysis
Conceptual mapping
Translation: reciprocal/refutational
Qualitative case description
Exploring relationships in the data
RELATIONSHIPS
ACROSS STUDIES
COMPARE
COMBINE
TRANSLATE
INTEGRATE
RELATIONSHIPS
WITHIN STUDIES
Exploring relationships in the data
RELATIONSHIPS
ACROSS
STUDIES
Mediators and moderators
Sub-set analysis
Conceptual mapping
Translation: reciprocal/ refutational
Qualitative case descriptions
RELATIONSHIPS
WITHIN STUDIES
Exploring relationships: tools



Moderators and mediators: categorising and viewing the
relationship between concepts according to their action on the
phenomena of interest
Sub-set analysis: Looking at actions, results or themes in groups,
dependent on characteristics
Visual representation:





Concept mapping: links between concepts are labelled and
placed in space to express a relationship
Ideas webbing: concepts organised in trees, hierarchies
Causal/cognitive mapping: representation aims to describe
direction of influences, key issues in real life
Translation: reciprocal/refutational: grouping and integrating
study findings in terms of level of agreement/fit in
argument/conclusions
Qualitative case description: writing a short narrative summary
across studies, concepts, themes, arguments
Output: Synthesis


Rigorous exploration of alternative interpretations of
relationships between and across studies
Some level of translation and integration with
explanation of (e.g.)





agreement/refutation
commonalities/outliers
order/groupings
priorities/emphasis/frequency
Identification of moderators (influencing factors),
mediators (mechanisms of action) and sub-groups
(components of programme theory)
EXERCISE: interpretation



Look at what you think might be the key
concepts/themes drawn from the findings of
the 3 studies
Play with the concepts/themes spatially to
explore the potential relationships and
patterns across concepts
Map the concepts/themes ACROSS
STUDIES to denote some aspect of the
relationship between them
Assessing robustness



Purpose: to assess the strength of evidence for
drawing conclusions about the results of the
synthesis in relation to methodological quality,
relevance, trustworthiness
Tasks: assess relationship of synthesis to original
review question, programme theory, study quality
and review method
Tools:






Assessing theoretical fit
Validity assessment
Weight of evidence
Critical reflection
Triangulation
Checking with authors
Assessing robustness
REVIEW QUESTIONS
PROGRAMME THEORY
Assessing robustness
REVIEW QUESTIONS
STUDY
QUALITY
Best Evidence Synthesis
Weight of evidence
Validity assessment
Critical reflection
Checking with authors
Triangulation
PROGRAMME THEORY
REVIEW
METHOD
Tools: assessing robustness






Best evidence synthesis: prioritising/choosing studies with the
strongest design
Weight of evidence: using a criterion of strength or value to interpret
the synthesis findings e.g. effect size, repeatability, harmony,
coherence
Use of validity/credibility assessment: using the results of critical
appraisal to interpret synthesis
Critical reflection: using the review author’s use of critical
tools/techniques/methods to consider results e.g. agreement,
alternative explanations, assessment of theoretical fit, parsimony,
elegance
Checking with authors: involving the authors of the primary studies in
comment, reflection, or building theory/synthesis
Triangulation: using the results of other data sources to compare the
results of the synthesis
Exercise

How do you think the methods and quality of
the original studies would affect the
interpretation of the findings?
Output: robustness




Identification of any limitations in the process of
review and synthesis
Assessment of the credibility and trustworthiness of
the synthesis product, seen through the
provenance, quality, and strength of the evidence
Assessment of the significance of the theoretical
contribution
Identification of its potential use and further
development
Review product suggestions

Formal report: PRISMA guidelines





method
analysis: description, interpretation
conclusion: theoretical contribution
implications
Practitioner report:




content,
examples, visual illustration
best practice guidance
tools, resource lists
Common problems








Volume of material
Level of material
Heterogeneity of material
Disaggregation
Crossing disciplinary boundaries
Lack of guidance on synthesis method
Time
Skill in original research methodology