Excellence in Qualitative Research Kathleen A. Knafl, PhD, FAAN Associate Dean for Research Frances Hill Fox Distinguished Professor The University of North Carolina at Chapel.
Download
Report
Transcript Excellence in Qualitative Research Kathleen A. Knafl, PhD, FAAN Associate Dean for Research Frances Hill Fox Distinguished Professor The University of North Carolina at Chapel.
Excellence in
Qualitative Research
Kathleen A. Knafl, PhD, FAAN
Associate Dean for Research
Frances Hill Fox Distinguished Professor
The University of North Carolina at Chapel Hill
Learning to Evaluate Qualitative
Research
Striving for quality as a researcher
Being evaluated – successful/unsuccessful
attempts to publish and secure funding
Evaluating others – proposal and
manuscript reviewer
Overview of Presentation
Nature and purpose of qualitative research
Position statements and guidelines for
evaluating qualitative research
Common expectations and indicators of
excellence
Overall Goal
Avoid an overly simplified or overly
complex discussion of issues
Present guidelines that address scientific
rigor while recognizing the distinct
qualities and contributions of qualitative
research
“Creativity must be preserved within
qualitative research, but not at the
expense of the quality of the science”
(Maxwell, 1990)
Nature and Purpose of Qualitative
Research
Multiple approaches
Terminological jungle
Ethnography
Phenomenology
Hermeneutics
Constructivism
Participatory action research
Grounded theory, etc…
Common Characteristics of
Qualitative Research
Focus on subjective perspective of
respondents
Emphasis on contextual understanding
Design flexibility; emergent design
Generation of narrative data
Researcher as primary data collection
instrument
Differences Across Approaches
Purpose
Philosophical underpinnings
Disciplinary roots
Theoretical orientation
Dominant methods for data collection
and analysis
Varying Purposes of Qualitative
Research
Description
Conceptualization and theoretical
understanding
Empowerment and social change
Description (e.g. descriptive phenomenology,
ethnography, qualitative description)
“The purpose of this study is to explore how
spousal carers of people with MS interpreted
their lived experience with their partner, the
way in they assigned meaning to their being
in such a situation, and the skills and
knowledge they have developed to live with
their situation” (Cheung & Hocking, 2004, p. 155).
Conceptualization
(e.g. grounded theory, concept development)
“The author investigated decision-making
experiences of 20 surrogates who assisted
terminally ill family members for this
grounded theory study. Findings describe a
basic social process of Seeing them Through
with Care and Respect” (Meeker, 2004, p.204).
Empowerment and Change
(e.g., feminist, participatory action research)
“In this participatory research study, injured
workers and academics together participated in
setting the agenda, determining the research
questions and methods, gathering the data, and
interpreting the results. The process helped all
understand more deeply the complicated reality of
being an injured worker within a set of system and
societal norms (Beardwood, et al., 2005, 33-344).
The Challenge of Evaluating
Qualitative Research
Evaluative stance – Should the same
criteria apply to all research? All qualitative
research?
Reasonable expectations – What should
we expect to see included in a qualitative
report/proposal?
Appropriate criteria – What standards
should we use to judge the merits of a
qualitative report/proposal?
Guidelines and Criteria
Guidelines for reporting (Tong, et.all, 2007)
Shared standards (Collingridge & Gantt, 2008;
Guba & Lincoln, 1985)
Alternative standards (Cheek, 2007; Cohen &
Crabtree, 2008; Davies & Dodd, 2002; Fossey, et
al., 2002; Whittemore, et al., 2001)
Shared Standards (Lincoln & Guba, 1985)
Truth Value (internal validity/credibility)
Applicability (external validity/
transferability)
Consistency (reliability/auditability)
Neutrality (objectivity/confirmability)
Qualitative
standard
Credibility (internal validity)
Defined as adequate representation of
group or situation studied
Quality of the data
Demonstrated by prolonged engagement,
persistent observation, triangulation,
external checks, negative case analysis,
member checking
Transferability (external validity)
Defined as the degree of similarity between
research site/participants and others
Demonstrated by thick description, reporting
of information for judgments about
transferability
Joint responsibility of researcher and
consumer
Dependability (reliability)
Defined as differentiating real
perspective/behavior of respondents from
reactive effects of research.
Demonstrated by audit of data collection
processes
Quality of interaction with subjects
External review of data
Confirmability (objectivity)
Defined as the adequacy of the results,
interpretations, and recommendations
Demonstrated by audit of analytic
procedures
Data processing and reduction
Data review and interpretation
External auditor
Alternative Standards (Davies & Dodd)
“Incorporating the notion of ethics, the authors
develop a cluster of terms around which they
argue that qualitative research can
meaningfully speak about rigor:
attentiveness, empathy, carefulness,
sensitivity, respect, reflection,
conscientiousness, engagement, awareness
and openness” (Davies & Dodd, 2002, p. 279)
Common Evaluative Criteria
(Cohen & Crabtree, 2008; Tong, et al., 2007;
Whittemore, 2001)
Importance of the research
Use of appropriate methods
Coherence of the research
Clarity of the presentation
Ethical conduct of research
Application of Common Criteria
to Research Reports/Proposals
Reasonable expectations
Indicators of excellence
Importance of Research
Reasonable expectations
General statement of purpose, research
questions, or aims
Review of relevant literature
Importance of Research
Indicators of excellence
Evidence of building on prior research;
addressing an important gap
Pragmatically or theoretically useful
Focus on the research problem, not the method
Making the Case for Importance
“The aim of this study was to identify approaches
primary care providers use to engage older men
in depression care. We focus specifically on
older men because previous work has shown
that they suffer from higher levels of undertreatment and that men differ from women in
gender-specific barriers to help seeking”
(Apesoa-Varano, et al., 2010, p. 587)
Use of Appropriate Methods –
Study Design
Reasonable expectations
Varied approaches
Emergent design
Indicators of excellence
Rationale for selection of approach;
fit with study purpose
Citations in support of the approach
Rationale for Study Design
“We chose grounded theory as the method for
data collection and analysis. It is a method for
conceptualizing patterns of behavior in which
people are engaged (Glaser,1978;1998). In
this study, the patterns of behavior are those in
which nurses engage while caring for palliative
cancer patients in hospitals” (Sandgren, et al., 2006,
p. 80).
Use of Appropriate Methods –
Role of Theory (Sandelowski, 2003)
Reasonable expectations
Variation in the source, function, and temporal
placement of theory
Different qualitative traditions specify different
roles of theory
Prevailing (though misleading) belief that
qualitative research is atheoretical
Use of Appropriate Methods –
Role of Theory
Indicators of excellence
Explicit statement of philosophical beliefs,
concepts, or theories that informed design
of study
When a concept/theory is an outcome of the
research, comparison to related concepts or
theories
Example - Use of Theory
“The philosophical framework for this study
was based on interpretive phenomenology”
(Cheung, 2004, 155).
“The purpose of this study was to examine
older African American women’s perceptions
of social and environmental stress in relation
to their heart disease through the lens of the
weathering conceptual framework” (WarrenFindlow, 2007, 234).
Sample Size and Selection
Reasonable expectations
Purposive approach; non-probability sample
Relatively small sample
Single case to 50+ participants
Likely range of number of participants
Sample Size and Selection
Indicators of excellence
Statement of purposive intent (e.g. maximum
variation, theoretical, intensity)
Specification of unit of interest
(individual, group, setting)
Rationale for sample size; invoking the principle
of saturation
Purposive Sample – Maximum
Variation
“The base line criterion for inclusion in the
sample was that all parents were employed and
had at least one child whom they identified as
disabled. The aim was to generate a sample that
included a range of situations in which parents
combined employment and care. This was not
intended to achieve generalizability, but to
enable examination of issues not related to a
specific homogenous group” (Lewis, et. al, 2000,
p.1035).
Purposive Sample – Theoretical
“ As data collection progressed, I conducted
theoretical sampling to provide data needed
to describe the categories thoroughly. For
example, because early participants were all
reporting that other family members had
been very supportive, I sought participants
who had experienced conflict (Meeker, 2004,
p.208) .
Data Collection –
Interviewing and Observation
Reasonable expectations
Use of single or multiple data collection
strategies
Description of:
• Setting for data collection
• Duration of data collection
• Identification of data collectors
• Description of observational or interview guides
• Method for recording data
Data Collection –
Interviewing and Observation
Indicators of excellence
Overall explicitness and thoroughness
• Description of role of the investigators; nature of
interactions with the participants
• Options for follow-up contact with participants
• Efforts to assure and monitor data quality
• Evidence of sensitivity to incoming data;
emergent design
Example – Interviews
Each participant was told, “I’d like you to tell me
the story of your experience with a chronic health
condition. Start at the time your symptoms began
and describe the things that happened one after
another regarding your health condition until
today. I encourage details because whatever is
important to you is of interest to me” (Lee & Poole,
2005, 349).
Example - Observation
“I collected data through participant observation
in two cardiac rehabilitation programs. In both,
I assisted in the day-to-day work of the staff,
attended education sessions, and conversed
with participants on an ongoing basis. When I
introduced myself to clients, I indicated I was
there to learn about heart disease from the
participants’ point of view and that I was carrying
out research (Wheatley, 2005, p. 440-441).
Data Processing and Analysis
Reasonable expectations
Overview of the steps the researcher took to
break the data into smaller units for the purpose
of analysis
Methodological citations that support the
approach
Variability in use of software programs to
support analysis
Data Processing and Analysis
Indicators of excellence
Analytic process is consistent with the
qualitative approach
• Line by line coding for grounded theory
• Extraction of significant statements for
phenomenology
The unit of study is preserved in the analysis
Evidence of thoroughness and checks
on quality
Explicitness
Results
Reasonable expectations
Variation in organizing structure for
presenting results
• Process
• Essential structure
• Thematic description
• Typology
Use of illustrative quotes or vignettes
Variable use of numbers
Results
Indicators of excellence
Vividness
• Compelling presentation
• A good read
• Creative
Coherence
• Integration of the data; more than a description
of codes and themes
• Convincingly addresses all study aims
Discussion
Reasonable expectations
Possibly integrated with results
Linkages to the body of knowledge in the field
Statement of applied and/or theoretical
implications of the findings
Limitations and next steps
Discussion
Indicators of excellence
Elaboration of pragmatic/theoretical implications
Limitations that address the unique aspects of
qualitative research (e.g. non-probability sample
not a limitation)
Explicit consideration of transferability
Ethics
Reasonable expectations
Statement that IRB approval has been obtained
Indicators of excellence
Evidence of sensitivity to human, social, and
cultural contexts
Recognition of ongoing ethical issues and
decisions
Overall Coherence and Consistency
Alignment across all aspects of the study
Completeness and follow through
Explicitness; thoroughness of presentation
Creativity – thoughtful, innovative use of
methods; insightful linkages
Balancing Rigor and Creativity
“We can preserve or kill the spirit of qualitative
work; we can soften our notion of rigor to
include playfulness, soulfulness, imagination,
and technique we associate with more artistic
endeavors, or we can further harden it by the
uncritical application of rules. The choice is
ours rigor or rigor mortis” (Sandelowski, 1993).
The University of North Carolina at Chapel Hill