Review

Professor Jan Illing Centre for Medical Education Research School of Medicine, Pharmacy and Health Durham University

Overview of the lecture

The role of the literature review in research Types of evidence: Logic of aggregation vs. logic of configuration • • • • • Different types of literature review Narrative Systematic Meta Analysis Realist synthesis Synthesising qualitative research

∂

Overview of the stages of a systematic review: research question, inclusion and exclusion criteria, searchers of title, abstract and papers, data extraction critical appraisal, assessing quality.

Reviews: Cochrane collaboration, Campbell collaboration, BEME

The role of the literature review

• • • • • • • Need to

understand the topic

and what is already known The review places research in the context of a body of literature Informs what still needs to be researched, before it is known

Learn about the methods used previously

– what methods are

∂

e.g. need for a qualitative approach

Avoids repeating

what is known, or repeating studies that have already failed.

Provides information on the

strength of available evidence

, quality of the studies and

identifies gaps

How does a literature review help you?

•

Helps to understand how one study is related to others

•

Highlights conflict in findings ∂

•

Can suggests new interpretations and new insights

•

Can identify what research is needed next

Different types of literature review answer different types of question ∂

Types of evidence

Aggregate

(counting evidence) vs.

Interpretive

(seeks understanding) Each study adds ‘weight’ Broadens or deepens understanding Studies add more weight Add insight to a pattern or emerging view with sample size, study design and Add theory strength of findings

∂

Reporting statistics Reporting themes

Types of review: Narrative

Narrative

– also called non-systematic, traditional and conventional

Provides a snap shot of prevalence knowledge

• • • •

Criticisms include

:

∂

The selection and inclusion of papers are not clear, Incomplete and sometime missing Risk of bias from selection of included studies Not easily to reproduce Conclusions are questionable from the quality of methodology used and quality of reporting

Systematic literature review

Unlike the narrative review, the systematic approach aims to search all (not just part) of the relevant literature in a systematic way, aiming to use – “

a reproducible method for identifying, evaluating, and synthesising the existing body of completed and recorded work produced by researchers, scholars, and practitioners

” (Fink 2005)

∂

Systematic reviews aim to inform decisions and need to be made on the best available evidence (Booth et al, 2012) Seen as ‘

a scientific activity

’ (Mulrow, 1995) Often continuously updated (Cochrane reviews)

Systematic literature review

Weaknesses

The criteria used to identify the studies can also be a source of bias or a limitation (Song et al., 2010) • • • Limited by year published e.g. last 10 years may miss important research published before this time

∂

Limited by language e.g. publication in English may focus on positive findings, publications in authors own language may report more negative findings Limiting searches to well-known databases can result in missing information. E.g. An audit of sources for a review of complex interventions (including qualitative evidence) found that only 30% were identified from databases and hand searches. About half of studies were identified by ‘snowballing’ and another 24% by personal knowledge or personal contact (Greenhalgh 2005).

What a review adds

• • Can help answer important questions that inform the value of an intervention, policy, practice, and identify what new research to do Can identify if finding tend to be consistent across a theme

Areas of conflict

∂

• • • • “

Disagreements among findings are valuable… conflict can teach us a lot

” Conflicts often identify important differences in sample and context May indicate a need for more research Could be used to illustrate a theory Highlight poor research methods.

Hierarchy of research evidence ∂

Stages of a systematic review

        Devise a research question Develop a protocol – objectives, search strategy, inclusion and exclusion criteria Search literature

∂

papers Search for relevant studies abstract, then papers Data extraction and critical appraise studies Synthesis of findings Interpretation Write up

A Scoping Review

Provides a snap shot on a topic

• The focus is on the range of studies that have been done • They may focus on an issue

∂

e.g. what theoretical approaches have been used to study x • There is no attempt to examine quality (Arksey and O’Malley, 2005)

Meta Analysis

Helps to answer questions about interventions when sample sizes need to be increased

• • • • • • Systematic search with inclusion/exclusion criteria Focus is to find studies on the same topic using the same study design e.g. RCT

∂

Combines results of quantitative studies Conducts statistical tests by combining data The analysis is numerical focus on the size of the effect

Realist synthesis

Realist synthesis is used mainly for the assessment of complex interventions

• • • • • • Starts with a systematic search of the literature Uses an interpretive approach, looking for patterns Aims to explain how outcomes occur

∂

Theory-driven approach Using evidence from qualitative and quantitative research methods The goal is to identify mid-range theories

Context Mechanism Outcome

The realist research question is often summarized as: "

What works for whom under what circumstances, how and why?"

Realist inquiry is based on a realist philosophy of science (reality can be known indirectly) and considers the interaction between

context, mechanism and outcome

∂

Each study is read and the reviewers seek to identify: the

contextual (C) influences

that are hypothesized to have

triggered the relevant mechanism

(s) (M) and generated the

outcome(s)

(O).

Known as CMOs

“

Education is a complex intervention which produces different outcomes in different circumstances

” (Wong et al 2012 Medical Education p89-96)

Development of theory

Realist synthesis looks for mechanisms and middle-range theories The aim is to test and build theory • • Mechanisms can be defined as: “…

underlying entities, processes, or structures which operate in particular

∂

” (Astbury and Leeuw, 2010) Mechanisms tend to generate outcomes, and are usually hidden and are sensitive to changes in context.

• Middle-range theories are developed from real data, so can be tested.

• Programme theory explains how the intervention is expected to reach its objectives.

Realist synthesis and secondary searches

Realist reviews start with a systematic review of the literature Assess the included papers and identify the CMOs and test out possible theories

Appraising the evidence

1. Relevance - can the study contribute to theory building or theory testing 2. Rigour - whether the method used to generate data is credible and trustworthy

Example: Realist synthesis of the literature on workplace bullying in the NHS

A systematic literature review focuses on research design and outcomes, but a realist review looks for patterns in context.

A realist review aims to explain why an intervention may work in one situation but not in another

∂

Findings from a realist synthesis focused on producing a theory to explain patterns in the data.

The findings from the study on workplace bullying highlighted that interventions were more likely to be successful if the senior managers were fully committed to them.

Illing, J, et al (2013). Evidence Synthesis on the occurrence, causes, consequences, prevention and management of bullying and harassing behaviours to inform decision-making in the NHS. HMSO.

Qualitative reviews

The focus is on understanding rather than effectiveness

The focus on effectiveness has privileged quantitative evidence over qualitative, and reduced consideration of other options to understand

∂

Complexity and context also need to be considered by practitioners and policy-makers

Searching for qualitative research

“

It is increasingly being recognized that evidence from qualitative studies that explore the experience of those involved in providing and receiving interventions, and studies evaluating factors that shape the implementation of interventions, have an important role in ensuring that systematic reviews are of maximum value to policy, practice and consumer decision-making

” (Mays 2005, Arai 2005, Popay 2005). MEDLINE (Wong 2004), CINAHL (Wilczynski 2007), PsycINFO (McKibbon 2006) and EMBASE (Walters 2006). However, identifying qualitative research is problematic due to the varied use of the term ‘qualitative’ (Grant 2004). Others argue for a purposive sampling approach, where searching is driven by the need to reach theoretical saturation and the identification of the ‘disconfirming case’, may be more appropriate (Dixon-Woods 2006).

Critical appraisal of studies

• • •

Weaknesses and flaws

Need to consider which are

fatal

There is no such thing as the and which are

non-critical

∂

and avoid

nihilism

(Sackett, 1997) Need to consider how much a weaknesses impacts on the findings

Right research design and method for the research question

Can the results be relied upon (valid, true) Are the results trustworthy, not just due to chance (reliable) Rigour – the results are not biased, confounding factors e.g. age and gender Limitations from selection bias, measurement of outcome, and analysis

Assessment of quality

• • • • • Quality and relevance check lists vary by discipline For novice researchers a check list can be helpful, Provides a list of everything that should be considered.

Aim to use check lists that are widely accepted or validated

∂

(Higgins and Green, 2008)

The problem with scales

The use of scales for assessing quality is discouraged in Cochrane reviews. Offers appealing simplicity,

not supported by evidence

(Emerson 1990, Schulz 1995b). A score involves assigning ‘weights’to items in the scale,

∂

difficult to justify the weights assigned

. Scales have been shown to be

unreliable assessments of validity

( Jüni 1999) They are

less transparent

to users of the review.

Preferable to

use simple approaches

for assessing validity that can be fully reported (i.e. how each trial was rated on each criterion)

Assessment of quality: in qualitative studies

Numerous tools are available to assess the quality of qualitative research (Cote, 2005,

Medical Teacher).

However, three structured appraisal approaches were compared and found that structured assessment tools may not produce greater consistency of judgements

∂

Journal of Health Services Research and Policy)

• • • • •

Check lists

Evidence for Policy and Practice Information (EPPI) Joanna Briggs Institute (JBI)

Web site: www.joannabriggs.edu.au/cqrmg

Spencer et al. 2003 conducted a review of many of the current appraisal frameworks and checklists, which may be helpful Critical Appraisal Skills Programme 2006 : h ttp://www.phru.nhs.uk/Pages/PHD/resources.htm

Expert judgement is also an important factor

•

Quality prompts in qualitative research

Are the aims and objectives clearly stated?

• • Is the research design clear and appropriate for the research question?

∂

Are the processes of obtaining the data clear?

• Is sufficient data provided to link interpretation and conclusions back to the data? • Is the method of analysis appropriate and adequately explained?

(Dixon-Woods et al, 2006, BMC Medical Research Methodology)

Quality indicators for all studies.

1. Is the research question(s) or hypothesis clearly stated?

2. Is the subject group appropriate for the study (number, characteristics)?

3. Are the methods used (qualitative/quantitative) reliable and valid for the research question and context?

4.

Completeness of ‘data’ Have subjects dropped out? attrition rate < 50%? 5. Control for confounding Have multiple factors/variables been removed or accounted for where possible?

∂

6. Are the statistical or other methods of analysis used appropriate?

7. Is it clear that the data justify the conclusions drawn?

8. Could the study be repeated by other researchers?

9. Does the study look forwards in time (prospective) rather than backwards?

10. Ethical issues, Were all relevant ethical issues addressed?

11. Triangulation, Were results supported by data from more than one source?

(Buckley et al, 2009, BEME review 11).

Collaborations

• Cochrane collaboration • Joanna Briggs Institute • Campbell collaboration, • Best Evidence Medical Education

∂

The Cochrane Collaboration

The Cochrane Collaboration was set up over 40 years ago by the epidemiologist Archie Cochrane.

The Cochrane Collaboration was set up to promote

∂

to synthesise data.

Evidence was rated strong or weak according to standard criteria which included research design and statistical effect size.

Helpful in making decisions about treatments – single variables, focused on outcomes.

Other approaches to systematic review

In Health

Joanna Briggs Institute ( http://www.joannabriggs.edu.au

) The central focus not only effectiveness, but also appropriateness, meaningfulness and feasibility of health practices and delivery methods. research evidence Also W K Kellogg Foundation ( http://www.wkkf.org

) Aim to improve health care practice, but do not place randomised controlled as high up in the hierarchy.

In Medical Education

Best Evidence Medical Education ( http://bemecollaboration.org

) Realist synthesis – RAMESUS project –what works, for whom, under what circumstances.

Campbell collaboration

• • • • The Campbell collaboration hosts reviews on: education, crime and justice, social welfare and international development.

∂

The Campbell Collaboration is an international research network that produces systematic reviews of the effects of social interventions. http:wwwcampbellcollaboration.org/

Best available evidence

Traditionally, the evidence based practice movement has focussed on the results of quantitative evidence (considering the

RCT as the gold

standard) to answer questions of effectiveness. Joanna Briggs Institute (JBI) has as its central focus not only effectiveness, but

∂

feasibility

of health practices and delivery methods. Best answered by considering other forms of evidence. The JBI regards the results of well-designed research studies grounded in any methodological position as providing more credible evidence than anecdotes or personal opinion. However, when

no research evidence

exists,

expert opinion

can be seen to represent the ”

best available

” evidence.

Conclusions

Literature reviews provide a range of purposes

, including knowing what has already been done and what needs to be done The approach used depends on the question and purpose of the review

There are a range of different types of review:

narrative, systematic, meta analysis, realist synthesis

∂

Most of these start with a systematic review

Assessment of

quality can challenging

, useful tools to support focus

Collaborations

Cochrane collaboration, Campbell collaboration, BEME effectiveness,

As well as effectiveness we need to think about using reviews to answer questions about appropriateness, meaningfulness and feasibility

Thank you Questions?

[email protected]

Centre for Medical Education Research School of Medicine, Pharmacy and Health Durham University