Transcript 10.55am Issues and Challenges around Searching the Literature

11.50am Issues and Challenges
around Appraising the Literature
Dr Chris Carroll, Senior Lecturer in
Health Technology Assessment, School
of Health and Related Research,
University of Sheffield
Overview
• What is Already Known on Appraisal of
Qualitative Research Literature
• Overview of Methodological Issues/Challenges
• Recent Developments (with focus on literature
of last two years)
• Outstanding Issues/Challenges
Key papers 1
• General:
– Hannes & Macaitis (2012). A move to more systematic and transparent
approaches in qualitative evidence synthesis: update on a review of
published papers, Qual Res, epub
• Criteria:
– Toye et al (2013). ‘Trying to pin down jelly’ – exploring intuitive
processes in quality assessment for meta-ethnography, BMC Med Res
Methodol, 13: 46
– Franzel et al (2013). How to locate and appraise qualitative research in
complementary and alternative medicine, BMC Comp Alt Med, 13: 125
– Gallacher et al (2013). Qualitative systematic reviews of treatment
burden in stroke, heart failure and diabetes- Methodological
challenges and solutions, BMC Med Res Methodol, 13: 10
– Garside (2013). Should be appraise the quality of qualitative research
reports for systematic reviews, and if so, how? Innovation: The
European Journal of Social Science Research, epub
Key papers 2
• Reporting:
– Garside (2013). Should be appraise the quality of qualitative research reports
for systematic reviews, and if so, how? Innovation: The European Journal of
Social Science Research, epub
– Tong et al (2012). Enhancing transparency in reporting the synthesis of
qualitative research (ENTREQ), BMC Med Res Methodol, 12: 181
– Tong et al (2007). Consolidated criteria for reporting qualitative research
(COREQ): a 32-item checklist for interviews and focus groups, Int J Qual Health
Care 19 (6): 349-357
– Salzmann-Erikson M (2013). IMPAD-22: A checklist for authors of qualitative
nursing research manuscripts, Nurse Education Today, epub
• Sensitivity analysis
– Boeije et al (2012). Making a difference: towards a method for weighing the
evidence in qualitative synthesis, Journal of Evaluation in Clinical Practice, 17:
657-63
– Carroll et al (2012). Should we exclude inadequately reported studies from
qualitative systematic reviews? An evaluation of sensitivity analyses in two
case study reviews, Qual Health Res, 22(10): 1425-34
Key papers 3
• Other literature
– Dixon-Woods et al (2007). ‘Appraising qualitative research for inclusion in
systematic reviews: a quantitative and qualitative comparison of three
methods, J Health Serv Res Policy, 12(1): 42-47
– Sparkes (2001). “Myth 94: Qualitative health researchers will agree about
validity”, Qual Health Res 11: 538-52
– Popay (2008). Using qualitative research to inform policy and practice. Cardiff;
ONS.
– Campbell R et al (2011). Evaluating meta-ethnography: systematic analysis and
synthesis of qualitative research, Health Technol Ass 15: 43
– Noyes & Popay (2007). Directly observed therapy and tuberculosis: how can a
systematic review of qualitative research contribute to improving services? A
qualitative meta-synthesis, J Adv Nurs. 2007 57(3):227-43.
– Morton et al (2010). The views of patients and carers in treatment decisionmaking for chronic kidney disease: systematic review and thematic synthesis
of qualitative studies, Brit Med J, 340: c112.
– Thomas & Harden (2008). Methods for the thematic synthesis of qualitative
research in systematic reviews, BMC Med Res Methodol, 8: 45
What is known
• That the issue is far from settled...
• Debate on whether critical appraisal of qualitative
research can or should be undertaken?
– Recognised need for it/it has a role; many feel that it needs
to be done; to know whether the research is “good
enough”, if it is to inform practice (Toye 2013)
– Often it is done: 59/82 reviews 2004-2008; only 5 of the
others stated explicitly why no critical appraisal was
conducted (Hannes 2012)
– So, as a consequence, “it becomes more important to shift
the academic debate from whether or not to appraise to
what criteria to use” (Hannes 2012)
What is known:
• If use is accepted…
– Debate on how to perform appraisal, which criteria to use
– Debate on how the findings of the appraisal to be used
• Why does it exist as an ongoing problem?
– Applying quantitative approaches to qualitative research;
– Philosophical and epistemological diversity of qualitative
research
• Neither issue is going away; neither has been
addressed adequately to date
Criteria:
• What criteria should you use for your review? What are you
trying to establish with the criteria?
– Is a study “good enough”?
• Researchers use multiple checklists and criteria, often
adapting existing checklists for their own review (Hannes
2012: 24 different tools in 59 reviews)
• Majority of papers cited in this presentation adapt tools or
means of assessment for their own reviews: Tong 2012;
Garside 2012; Franzel 2013; Gallacher 2013 etc.
• There appear to be no “standard” criteria; mostly the
choice is left to the reviewers
Tools and criteria:
• This is in accordance with Cochrane and others’
approaches – use whatever you think is best ...
– “An instrument underpinned by a multi-dimensional
concept of quality in research ... According to several
domains ..including reporting, methodological rigour and
conceptual depth and breadth ... Reviewers need to decide
which instrument appears to be most appropriate in the
context of their review” (Franzel 2013; Ch.4 of
Supplementary Guidance for Cochrane Reviews 2011)
– Toye 2013: [“the criteria by which we judge quality are not
fixed but shift and change over time in relation to
context”]
– Garside 2013: Criteria depend on perspective (uses
Sparkes’ 2001 typology of validity to articulate the various
perspectives)
Criteria:
• Garside acknowledges relevance of Sparkes’ 2001
typology of validity, which encompasses all
perspectives:
– Standardisation for all research (replication perspective);
– one distinct to qualitative research (parallel perspective);
– one each for all of the different types and approaches to
qualitative research (diversification of meanings
perspective);
– the “no appraisal” approach (Letting go of validity)
• Mostly, choice is left to reviewers
Criteria:
• Most reviewers use/adapt existing tools. What are the
key/core elements?
• Domains
– Reporting, methodological rigour and conceptual depth
and breadth (Cochrane; Franzel 2013)
– Study conduct; reporting; theoretical and interpretive
validity (Hannes 2012)
– Study conduct; reporting; implicit judgement of content
and utility of findings for theory development (Tong 2012)
– Technical processes (reporting and conduct of study); and
trustworthiness and theoretical and practical
considerations (Popay 2008; Garside 2012)
• Very broad … what does it look like in practice?
Franzel (2013) criteria:
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
Is question clearly formulated
Which design was selected and why?
Was a literature review performed?
Were participants and researchers and environment sufficiently described?
Was data collection method described in detail?
Was data analysis method described in detail?
Was data saturation reached?
Are results comprehensive, plausible and coherent?
Were the results confirmed?
Do results apply to my own practice?
Can results be transferred to other settings?
12. Openness
13. Intersubjective validity
14. Reflexivity
Toye (2013) criteria:
1. Reflexivity: Who the researcher is affects the
interpretations
2. Ethical relationship between researcher and
participant
3. Who are the participants; who is not there
4. Is the approach inductive and grounded in the data: is
there a clear link between the proposed concepts and
the data used to illustrate them?
5. Has researcher looked for and discussed contradictory
data? Are interpretations being challenged?
Gallacher (2013) criteria:
Appraisal Question (apply each question to the whole study to reach an overall conclusion i.e. aims, sampling, data
collection, data analysis, interpretations)
Does the research, as reported, illuminate the subjective meaning, actions, and context of those being researched?
Are subjective perceptions and experiences treated as knowledge in their own right?
Is there evidence of the adaption and responsiveness of the research design to the circumstances and issues of real-life social
settings met during the course of the study?
Does the sample produce the type of knowledge necessary to understand the structures and processes within which the
individuals or situations are located?
Is the description provided detailed enough to allow the researcher or reader to interpret the meaning and context of what is
being researched?
Are any different sources of knowledge about the same issue compared and contrasted and how is this done?
Has the researcher rendered transparent the processes by which data have been collected, analyzed, and presented?
Has the researcher made clear their own possible influence on the data?
Is it clear how the research moves from a description of the data, through quotation or examples, to an analysis and
interpretation of the meaning and significance of it?
Are claims being made for the generalizability of the findings to either other bodies of knowledge or to other populations or
groups and if so what are these claims?
Is there any other aspect of the study that may affect the quality e.g. conflict of interest?
Criteria:
• Appear to be specific criteria depending on
review, its question and perspectives of the
authors
• But does this approach work?
Issues with criteria:
• Franzel (2013): Criteria on openness, reflexivity etc. rarely satisfied;
poor reporting
• Toye (2013): “’You end-up measuring what you can measure rather
than what is necessarily important’
• So, can choose your criteria, but hit the barrier of reporting
• Poor reporting is endemic; can only begin to make judgements
about technical processes, conduct and trustworthiness etc., when
you have a clear picture of what was done: Carroll 2012; Hannes
2012
• Therefore:
– Standardise the reporting of “what is important”
– COREQ (Tong 2007), ENTREQ (Tong 2012), IMPAD-22; Garside 2012
Other barriers …
• Whether or not a study (as published) satisfies
any given criteria (with consequences for the
use of the appraisal in any synthesis) can be
dependent on (Garside 2012):
– The requirements being made of the study’s
authors: by the journal, by their discipline, by their
research (a short piece formulating a single
specific concepts or a long complex piece)
– The fit between the primary study’s question and
that posed by the review
Expert opinion rather than checklists:
• What about a more intuitive assessment of quality
rather than strict criteria: Toye (2013)
– KP: Key paper; SAT: Satisfactory paper; FF: Fatally flawed
– “Early in the process of appraisal we found that the
categories ‘KP’ and ‘FF’ were not mutually exclusive”
– “’Being in this team made me realise that I am drawn in by
the readability and seductiveness of a research piece’”
– “’There is a lethal cocktail of poor methodological study
with a lack of interpretation that is incredibly well-written
... It can be incredibly emotive and instil lots of passion’”
– “The process challenged our ‘intuitive certainty’ and was
the focus of knowledge reconstruction”
– (cc. Dixon-Woods 2007)
An alternative?
• Method-specific criteria:
• Appear to be criteria developing for particular
approaches:
– Best-fit framework synthesis: Four criteria on
quality of reporting
– Realist synthesis: Criteria of utility and relevance
Summary on criteria and appraisal
process:
• Some approaches specify criteria;
• Some approaches advocate criteria selection for each
review based on context (Cochrane);
• Most methods encounter the barrier of reporting;
• Generally there is inconsistency in conducting the
process, using whichever criteria being applied, so
critical consensus is recommended (Garside 2012; Tong
2012)
• Therefore, still no resolution, but problem is being
addressed
How is it to be used?
• So, have different approaches ... but all studies
that consider value of appraisal also consider
value of process to the synthesis
• “it is through doing synthesis that the critical
elements of quality are illuminated” (Garside
2012)
• Exclusion of papers deemed “not good
enough” appears to be the principal reason
for appraisal (Hannes 2012: 30 of 59 reviews)
Threshold
• However, what is the threshold for exclusion (“rationale for
weighting or excluding … should be explicit”: Tong 2012):
– Franzel (2013): Not satisfying any criteria
– Carroll (2012): Failing to satisfy no more than 1 of the 4 criteria
– Toye (2013): “we excluded studies from our meta-ethnography
on the basis that the methodological report was insufficient to
make a judgment on interpretive rigour ... (by consensus on
intuitive criteria)”
– Noyes & Popay (2007) - judging conceptually: “thickness”;
Thomas & Harden (2008) – failure to satisfy all seven criteria
• But can you, should you exclude?
Options if not just excluding: Weighting by
thresholds
Boeije 2012:
• Multiple methods: scores achieved by checklist and expert
judgement, gave more weight to certain findings because the
studies from which certain themes emerged scored relatively
more highly, even if other themes appeared more frequently;
however, generally low frequency themes appeared in
relatively weaker studies, “an indication that these concepts
need further research that is methodologically sound”
• “The outcomes of this method could be compared to
sensitivity analysis … the question to be answered is, what
would happen to the results if studies below a certain
established ‘quality threshold’ would systematically be
excluded?”
Options if not just excluding: Sensitivity
analysis
• Include all studies to test their impact on the synthesis:
– Carroll (2012): “There is correlation between quality of
reporting of a study as its value as a source for the final
synthesis ... There is a possible case for excluding
inadequately reported studies”
– Franzel (2013): “The themes from the excluded papers
would not have altered the meta-synthesis. Theoretical
study saturation was achieved because no new relevant
data emerged regarding a category, either to extend or to
contradict it”
– Tong (2012): “studies with sparse detail about the conduct
of research tend to contribute less to the synthesis”
– Morton (2010); Noyes & Popay (2007), Thomas & Harden
(2008)
But …
• Garside (2012): “since the findings were similar
[between poor and high quality studies], one could
conclude that there was no fatal flaw in the design or
execution of the … “poor quality” research reports.
Their inclusion may have strengthened the evidence
for particular findings in the synthesis, especially if they
had emanated from different disciplines or used
different methods”
• Carroll (2012) reported that the exclusion of
inadequately-reported studies would have seen the
exclusion of a large group of nursing studies, thus
minimising transferability of the findings
• Sensitivity analysis rather than outright exclusion might
therefore offer a more worthwhile approach
If not excluding, weighing or analysing, any
point in doing appraisal?
NO
• Campbell (2011): “The inclusion of poorer studies
is unlikely to have a very distorting impact on
qualitative synthesis ... [the time spent on quality
appraisal might perhaps be better spent]”
• Toye (2013): “Although we excluded studies from
our meta-ethnography on the basis that the
methodological report was insufficient to make a
judgment on interpretive rigour ... The process of
appraisal was extremely time consuming, and we
also need to ask whether or not this is time well
spent”
If not excluding, weighing or analysing,
any point in doing appraisal?
NO
• If not excluding because “no scoring system or level of pass
mark was set as the value of this is uncertain”: Gallacher
(2013)
• Gallacher (2013): “Appraisal was not carried out to exclude
studies but to inform the discussion ... To highlight any
notable defects in the quality of the literature ... Papers
were generally of reasonable quality [but] ...
Acknowledgement of researchers influence on the analysis
... [was less well demonstrated] ... The value of carrying out
quality appraisal in this review is debatable ... This supports
our decision not to exclude ... but raises the issue of
whether quality appraisal under these circumstances is a
worthwhile process”
If not excluding, weighing or analysing,
any point in doing appraisal?
YES
• Tong (2012): “To allow readers to make an informed
judgment about the credibility … dependability …
transferability and confirmability (are research findings
and interpretations linked to the data?) … of findings …
as a screening tool for future studies… highlighting
effective methods … to facilitate a deeper
understanding of the included papers”
• Checklists can make judgments transparent for team
members and others, even if they are limited to what is
reported and the findings are scarcely used in the
synthesis
Summary on quality assessment and
synthesis:
• Subjectivity of process and relative value of
criteria present problems for using findings in
the synthesis
• Most frequent action:
– Exclusion
• Increasingly recommended action:
– Sensitivity analysis
• Alternatives seem to be of questionable worth
Conclusions
• Variety and subjectivity of appraisal tools and
criteria present a problem for multiple
stakeholders: those conducting, critiquing and
seeking to use QES, yet is still not resolved;
• Are there means of accommodating variety and
appropriateness, while making the process
rational, transparent, appropriate and useful?
• Link to synthesis offers the most worthwhile
perspective for addressing the problem in future
research
Conclusions
• There may be a role for specific types of appraisal
to accompany specific methods of review and
synthesis
• Otherwise options might be ...
– Not to appraise at all because it is too subjective,
time-consuming and of little demonstrable value to a
synthesis (but this is an old view);
– To exclude studies using specific criteria or, better still,
to conduct sensitivity analysis
• However, problems remain over appropriateness
and weighting of criteria, even if the role of
appraisal in synthesis is being clarified
Conclusions
• Toye et al (2013): “If we want to use qualitative
research to inform clinical practice, we need to be
confident that the research is good enough ... It is not
surprising that ... researchers do not agree about
whether or not to exclude studies on the basis of
quality, as, over an above any epistemological
arguments about multiple realities, there is no
agreement about what quality is ...Qualitative
researchers cannot ignore the debate about quality
and their intuitive certainty should not remain
sacrosanct... It is not good enough to say that we know
quality when we see it”.