Transcript Lecture 7: Judging Qualitative Research

Judging
Qualitative
Research
The Role of the Reader

"There are no operationally defined truth tests to apply to
qualitative research" (Eisner, 1991, p. 53).

Researcher and readers "share a joint responsibility" for
establishing the value of the qualitative research product
(Glaser and Strauss, 1967, p. 232).

"Pragmatic validation [of qualitative research] means that the
perspective presented is judged by its relevance to and use
by those to whom it is presented: their perspective and
actions joined to the [researcher’s] perspective and actions"
(Patton, 1990, p. 485).

Validity of research corresponds to degree to
which it is accepted as sound, legitimate and
authoritative by people with an interest in
research findings.

How do we judge which perspective to use to
evaluate the validity of a qualitative study?

E.g., grounded theory should theoretically sample a wide
range of people; discourse analysis can be in-depth
analysis of a few excerpts

Process of agreeing criteria for judging qualitative
research is useful because involves critically
reflecting on essential ingredients & practices

But simply following guidelines does not
guarantee good research – not simply a
descriptive science but relies on capacity to evoke
imaginative experience & reveal new meanings.
3 features

Coherence

Consensus
(structural corroboration or
triangulation)
Instrumental Utility
"Guides call our attention to aspects of the situation
or place we might otherwise miss" (Eisner, 1991,
p. 59

Trustworthiness
"How can an inquirer persuade his or her audiences
that the research findings of an inquiry are worth
paying attention to?" (Lincoln and Guba 1985, p.
290).
Lincoln and Guba (1985, p. 300) - an alternative set
of criteria that correspond to those typically
employed to judge quantitative work.
Comparison of criteria
Conventional terms
Internal validity
External validity
Reliability
Objectivity
Naturalistic terms
Credibility
Transferability
Dependability
Confirmability
Critical of use of "comparable criteria”

little different than the conventional criteria

assumes that what is known — existent or interpreted reality
— stands independent of the inquirer and can be described
without distortion by the inquirer

naturalistic research can offer only an interpretation of the
interpretations of others

to assume an independent reality is unacceptable for many
qualitative researchers
e.g., Smith & Heshusius 1986

there is no "out there" out there: the only reality is
a completely mind-dependent reality, which will
vary from individual to individual;

therefore, not possible to choose a best
interpretation from among the many available,
because no technique or interpretation can be
"epistemologically privileged" (p. 9).

This stance prohibits the possibility of reconciling
alternative interpretations

Important to determine which criteria are
consistent with the naturalistic paradigm, yet
which allow for a declaration that good research
has been carried out.

Select appropriate criteria for judging overall
trustworthiness of a qualitative study
Internal Validity vs. Credibility
Conventional inquiry
Naturalistic inquiry
= Extent to which findings
accurately describe reality.
= Assume the presence of multiple
realities and try to represent these
multiple realities adequately.
Must postulate relationships and
then test them.
Postulate cannot be proved, only
falsified.
Isomorphism is in principle
impossible: if we knew precise
nature of reality, no need to test it.
Credibility becomes the test for
this.
Credibility-A ‘toolbox’ of procedures
for enhancing validity
Triangulation: Enrich understanding of a phenomenon by
viewing from different perspectives.
4 types:
1) methods triangulation;
2) data triangulation;
3) triangulation through multiple analysts;
4) theory triangulation.

gather data from different groups of people; gather data at
different times from same people; different theories/methods
‘composite analysis’; triangulate perspectives of different
researchers.

Comparing researchers’ coding


Inter-rater comparison of subtle, complex coding
schemes; participant feedback/respondent
validation
Making segments of the raw data
available for others to analyze, and the
use of "member checks," in which
respondents are asked to corroborate
findings.

Disconfirming case analysis


Deviant or negative cases, systematically
searching for data that does not fit themes &
patterns
Audit (paper) trail

Provide evidence linking raw data to final report
Audit trail
Consisting of
1)
raw data;
2)
analysis notes;
3)
reconstruction and synthesis products;
4)
process notes;
5)
personal notes; and
6)
preliminary developmental information
*Critical component = conceptual chain of logic - mimics
replicability process in conventional research.
External Validity / Generalizability versus
Transferability
Conventional: seek findings which Naturalistic: aspire to theoretical,
are statistically generalisable
vertical or logical generalization of
findings
= Ability to generalize findings
across different settings.
Involves a trade-off between
internal & external validity (can
incl. only limited aspects of each
local context).
= transferability of a working
hypothesis to other situations
depends on the degree of
similarity between the original
situation and the situation to which
it is transferred.
Existence of local conditions
makes it impossible to generalize.
Reliability versus Dependability
Conventional Inquiry
Naturalistic Inquiry
(Focus instead on achieving greater
validity in work)
=
1) the degree to which a
measurement, given repeatedly,
remains the same;
2) the stability of a measurement over
time;
3) the similarity of measurements
within a given time period
= use an inquiry audit in which
reviewers examine both the process
and the product of the research for
consistency.
"Since there can be no validity without
reliability (& thus no credibility without
dependability), a demonstration of the
former is sufficient to establish the
latter" (Lincoln & Guba 1985: 316).
Objectivity versus Confirmability
Conventional Inquiry
Naturalistic Inquiry
Relies on quantitative measures
to define a situation is relatively
value-free, & therefore objective.
Relies on interpretations and is
admittedly value-bound, is
therefore considered to be
subjective.
Subjectivity leads to unreliable &
invalid results.
Question the possibility of ever
attaining pure objectivity.
Empathic Neutrality

Patton (1990): the terms objectivity and subjectivity have
become "ideological ammunition in the paradigms
debate." Use instead "empathic neutrality" (p. 55).


Empathy "is a stance toward the people one encounters, while
neutrality is a stance toward the findings" (p. 58). A researcher
who is neutral tries to be non-judgmental, and strives to report
what is found in a balanced way.
Lincoln and Guba (1985): the "confirmability" of the
research - the degree to which the researcher can
demonstrate the neutrality of the research
interpretations, through a "confirmability audit."
Demonstrating validity

Sensitivity to context: allow patterns &
meanings NOT prespecified to emerge





Relevant theoretical & empirical literature
Socio-cultural setting
Participants’ perspectives
Ethical issues
Empirical data
Sensitivity to the context
of existing theory &
research in the
development of a
research topic
Study lacks features of
validity
Study demonstrates features
of validity
Study sets out to simply
‘explore women’s
experiences of postnatal
depression’, ignoring
relevant theory & previous
qualitative studies of
experiences of maternity &
depression in women
Study clarifies what is already
known form theory & research,
formulates a specific research
question that has not been
addressed: “How does the
family context influence
women’s experiences
following childbirth?”
Sensitivity to how the
perspective & position of
participants may influence
whether they feel able to
take part & express
themselves freely
Senior male professional
carries out interviews with
women with postnatal
depression in a clinic
setting, ignoring possibility
they may feel less able to
express feelings to a man,
my be intimidated by clinical
environment.
Participants give choice of
taking part in focus groups
(allowing solidarity with
pother women in similar
experiences) or interviews
in own home (privacy,
security, accessibility)
carried out by women of
own age.

Commitment & rigour




through data collection
Depth/breadth of analysis
methodological competence/skill
In-depth engagement with topic
Commitment & rigour in
recruitment of participants
who will represent an
adequate range of views
relevant to research topic
Sample comprises 12 selfselected volunteers. Most
are well-educated, affluent,
married.
12 women are purposively
sampled to include married,
co-habiting & single
participants from affluent
and socio-economically
disadvantaged
backgrounds.

Coherence & transparency




clarity & power of argument
fit between theory and method
transparent methods & data presentation
Reflexivity
Transparency in the
analysis of data
Little description is provided
of how themes were
identified and no checks on
their consistency reported
A detailed description is
provided of how data were
initially coded and how
codes were modified
through comparison of all
instances and discussions
between the researchers.
Coherence between the
qualitative design & the
analysis and presentation of
data
Based on a frequency count
of the occurrence of codes,
the researcher highlights
the finding that 2/3 of the
single women (n=3) but
only 1/3 of married women
complain of lack of social
support. Strong
quantitative statements of
this kind are inappropriate
given small sample size
and reliability of codes is
unknown.
Based on a qualitative
comparisons of all
instances of the codes, the
researchers note that there
was a tendency for single
women to report a lack of
social support. However
disconfirming instances are
discussed as revealing
examples of why married &
cohabiting women may feel
unsupported and how
single mothers may be
supported by others.

Impact and Importance



practical/applied
theoretical
socio-cultural
Impact of the research
The researcher simply
notes that the findings are
compatible with existing
models and research (e.g.,
showing that single mothers
feel they have less social
support).
The researchers explain
how different family
structures and relationships
may exert positive or
negative influences on the
experience of maternity,
suggesting questions for
further research and ways
of identifying and
supporting women at risk of
depression.

Not always possible to meet some criteria,
sometimes must prioritize some kinds of
validity over others

A set of principles to refer to when making
decisions about how to carry out & to justify
your research.
Reading
Willig Chapter 9.
Barbour, R.S. (2001) Checklists for improving rigour in qualitative research: a case of
the tail wagging the dog? British Medical Journal, 322:1115-1117.
Cho, J. & Trent, A. (2006) Validity in qualitative research revisited. Qualitative
Research, 6, 319-340.
Harré, R. (2004) Staking our claim for qualitative psychology as science. Qualitative
Research in Psychology, 1, 3-14.
Henwood, K. (2004) Reinventing validity. In Todd, Z., Nerlich, B., McKeown, S. &
Clarke, D.D. (Eds.). Mixing methods in Psychology: The Integration of Qualitative
and Quantitative Methods in Theory and Practice. Psychology Press. Chapter 3.
Parker, I. (2005) Qualitative Psychology: Introducing Radical Research. OUP. Chapter
10.