Transcript Qualitative Mixed Methods and Design Issues

Science is the belief in
the ignorance of the experts
Richard Feynman, 1965 Nobel
Laureate, Physics
Mark Luborsky
MCUAAAR workshop 5/12
1
Qualitative Mixed Methods
And Design Issues
Mark R. Luborsky, PhD
Institute of Gerontology
Wayne State University, Detroit, MI
MICHIGAN CTR. FOR URBAN AFRICAN AMERICAN AGING RESEARCH
Please do not duplicate or use these slides without the express permission of the author.
• “Social science is not a science… they have not yet found out
anything, they haven’t got anywhere yet, maybe someday…”
• “If the theory is right, what should happen, and has that happened
needs study… in order to do that I must first understand how the
answer probably looks… get a qualitative idea of the phenomena
before I could get a good quantitative idea… so I have been working
with the hope that in the future that rough understanding can be
refined into quantified...”
1999, “The Pleasure of Findings Things Out”, Richard Feynman
Mark Luborsky
MCUAAAR workshop 5/12
3
Ascidians (“sea squirts”) are
hermaphrodite marine
invertebrate and exhibit traits
of the earliest ancestors of
chordates and vertebrates.
… but with a difference
Topics and Take Aways
m-m  M-M
Formulating best practices for M-M
(vs bashing disciplines or any
particular project or researcher)
I. Concepts and considerations
• Mixed discourse, multiple methods, Mixed Methods
• Conceptual heart of project
• Crafting the place of your basic scientific question in the body of
knowledge, not the doing (eg aims as: examining, exploring, analysing)
mechanical application of sets of procedures and methods
• II. Design considerations
• III. Succeeding in review and publication
Your take aways:
• Questions questions questions and concepts
• Examples to think with
• Pragmatic skills, vocabulary and guidelines
Mark Luborsky
MCUAAAR workshop 5/12
5
Mark Luborsky
MCUAAAR workshop 5/12
6
Mark Luborsky
MCUAAAR workshop 5/12
7
Envisioning new horizons
… the ideal….
•
Rich productive
collaboration stands on
firm disciplinary
strengths
•
and steps into the shared
margins to discover new
opportunities
Mark Luborsky
MCUAAAR workshop 5/12
Multi-disciplinary Collaborative Research
…
closer to the reality
8
Mark Luborsky
MCUAAAR workshop 5/12
9
Is it true you can now get funding …..
•
•
•
For a qualitative and mixed methods research project?
. . . NO
More easily for epidemiology, physics, and clinical trials?
. . . NO
A study of
 an important question,
 grounded in the scholarship on the topic, with a
 full and clearly developed design and methods that fit the
aims required to move us forward beyond what we already
know.
. . . YES to good scholarship
work to add to our body of knowledge about
factors shaping health and effective human
functioning in individuals and groups
Mark Luborsky
MCUAAAR workshop 5/12
10
Bio-monitoring of Persistent Toxic Substances in Michigan 
Lower-Income Urban Fish Eaters: Venue Based Sampling
of Shoreline Anglers Fishing Lower Tittabawassee,
Saginaw, and Detroit Rivers” [CDC/EPA/MDCH/ATSDR]

Improving community awareness for Detroit River Fish
Advisories [Erb Foundation]
“Developing Meaningful Life: Social Reintegration of
Service Members & Veterans with Spinal Cord Injury”
[Dept of Defense (DoD) / US Army CDMRP]
M-M. From the
laboratory bench to
the dinner table.
Multiple concurrent
methods


QT  QL
Conjoint
standardized & QL
“Using a System-Wide Database to Reduce Workplace
Violence in Hospitals” [ DHHS/CDC/NIOSH]

QT/Epi > QL
“The Meaning of Self-rated Health” (NIH/NIA



QT/Epi < QL
Conjoint QT & QL
QT  QL
QL  QT
“Downsizing For Household Relocation in Later Life”
[NIH/NIA]
State-wide survey Michigan Older Adults Relocatoin
HRS supplement questions 2011
Mark Luborsky
MCUAAAR workshop 5/12
11
Secret Lives:
Moral Aspects of Core Scientific Constructs for
Design & Analyses
•
•
•
•
•
•
•
•
Power
Validity
Reliability
Sample/Sampling
Representation/representative
Control
Dependent variable
Independent variable
The outside “second life” in daily society and talk connotations of these uses bring in enduring cultural dilemmas and
challenges for scientists working to achieving adequacy for each of those constructs
Luborsky, M. & R. Rubinstein. 1995 Sampling in Qualitative Research: Rationale, Issues, Methods
Research on Aging 17(1):89-113
Luborsky, M. 1994 Identifying Themes and Patterns. In J. Gubrium, & A. Sankar (eds) Qualitative
Methods in Aging Research. NY: SAGE
Mark Luborsky
MCUAAAR workshop 5/12
12
Mixed discourse, m-m, M-M
Two examples of pervasive substantial confounds
Identification of themes and patterns (Luborsky 1994)
• Themes
• Patterns
• Focus group analysis (Agar & Hobbes 1996)
• Themes
• Negotiation of relationships and values
• Consider: William Dressler, arterial hypertension in AA,
Mark Luborsky
MCUAAAR workshop 5/12
13
14
“Self rated health” powerful predictor of
mortality in later life across the globe.
But, what's this really about
code = “Physical Body Only” (Idler & Benyamini 2002)
[Original] Eleven years ago I had a mastectomy. I’ve had a very bad
case of high blood pressure for many years. For 15 years, I’ve had
diabetes. I have a sciatic nerve problem, which causes my legs to hurt
when I walk. The last doctor I went to wanted to do surgery to remove
the nerves around my spine. I refused and said I’d come back when I
couldn’t walk at all. He said if I didn’t want surgery, he couldn’t help
me.
< ‘chunked’ text to reveal narrative traits (evaluation) and unmet social contract >
1.
2.
3.
4.
Eleven years ago I had a mastectomy.
I’ve had a very bad case of high blood pressure for many years.
For 15 years, I’ve had diabetes.
I have a sciatic nerve problem, which causes my legs to hurt when I
walk.
5. The last doctor I went to wanted to do surgery to remove the nerves
around my spine.
6. I refused and said I’d come back when I couldn’t walk at all.
7. He said if I didn’t want surgery, he couldn’t help me.
Mark Luborsky
MCUAAAR workshop
15
[‘chunked’ text reveals narrative traits (evaluation) and unmet social contract]
a) Eleven years ago I had a mastectomy.
b) I’ve had a very bad case of high blood pressure for many
years.
c) For 15 years, I’ve had diabetes.
d) I have a sciatic nerve problem, which causes my legs to hurt
when I walk.
e) The last doctor I went to wanted to do surgery to remove
the nerves around my spine.
f) I refused and said I’d come back when I couldn’t walk at
all.
g) He said if I didn’t want surgery, he couldn’t help me.
Mark Luborsky
MCUAAAR workshop
16
Mark Luborsky
MCUAAAR workshop
17
What is the all noise about
qualitative and mixed methods?
DNA mapping
Socio-cultural distribution of
• forms of knowledge construction, and
Mark Luborsky MCUAAAR workshop
18
• of disciplinary
‘ethnic’ tension
about it
5/12
Concepts and Considerations
• History not new. Stouffer et al. “The American Soldier” 1940s
•
•
•
•
(Likert, etc)
Social science M-M is affliction not in other fields
inquiry/discovery
Compatibility and contentions between paradigms within a
field, tensions between discipline tensions (NIA/NINR review).
Motivation – trendiness will not carry your. Establish scientific
need, acknolwedge plus and minuses and address
Just because you can: quantify qualitative, or qualitative
describe experience -- doesn’t mean it will adequately
capture the construct or provide an real scientific impact
Mark Luborsky
MCUAAAR workshop 5/12
19
Is your project or article ready for review?
Mark Luborsky
MCUAAAR workshop 5/12
20
General Considerations
• No one preferred. Help reviewers see
multiplicity of approaches
• Some are more palatable to the NIH’s
positivistic quantitative approach
• Review panels less biased against a method
than incompletely specified proposals
• NIH is a public health agency: it is accountable
for improving public health
• Researcher bears burden of proof regarding
accountability and usefulness of methods
Mark Luborsky MCUAAAR workshop 5/12
21
What criteria do I use as editor of MAQ and NIH reviewer
• I go to the “methods” section to see if quantitative and qualitative data are
•
•
•
•
•
•
collected.
I look in the “analyses” “results” and “discussion” sections for some
connection, merging, or embedding of one type of data in the other.
I look in the “methods” discussion for rigorous quantitative and qualitative
procedures.
I look throughout the manuscript for how the author has positioned the
study within the field of mixed methods research and how the study
contributes a thoughtful, unique contribution to mixed method literature.
I look in the “background and significance” and throughout the manuscript
for the paradigm stance of the author.
I look at the references to see if the author cites and is familiar with the
mixed methods literature.
I look in the “introduction” for the rationale for using mixed methods to
study the research problem.
•
•
Mark Luborsky
MCUAAAR workshop 5/12
22
Mark Luborsky
MCUAAAR workshop 5/12
23
A few journals dedicated to mixed methods …
• Journal of Mixed Methods
• Field Methods
• Quality and Quantity
• International Journal of Multiple Research Approaches
(on-line)
• Other journals that publish mixed methods studies
(e.g., International Journal of Social Research
Methodology)
• Special issue journals
Practice Targets for Qualitative
M-M Applications to NIH
• INTERNAL CONSISTENCY!! Across problem, aims,
literature, design, methods
• Clarity (simple), specificity, and justification for
methods are key
• “Watch your language” especially shared terms
• Attend to sampling, sample collection and
maintenance
• Design. Define, then explore strengths and limits
• Mixed methods. How are they combined in process
and final product
• Budget for expertise, resources, time: add if needed
• Anticipate questions and answer them in writing
Mark Luborsky MCUAAAR workshop 5/12
24
14 Common Limitations in Proposals
With Qualitative Methods
• Aims not specific, err by stating procedures (e.g conduct
•
•
•
•
•
•
ethnography, do interviews, observe…)
No compelling answer to the “So what” question, need to
address the significance of proposed findings
Adversarial and dogmatic vs informed and reasoned
discussion about choice for this particular topic and study
Significance too general, lacks detail on specific factors
and topics. Over-developed “Background” relevance
State of knowledge used to structure proposal biased by
only covering studies using your methods
Methods not well justified: missing pieces, incomplete
descriptions - especially “measures”
Methods are internally inconsistent
Mark Luborsky MCUAAAR workshop 5/12
25
14 Common Limitations
(continued)
• Limitations of procedures not address (sample, data
collection, analyses)
• Sampling decision not explained with sound rationale
• Data collection procedures not explained in
adequate detail (when, where, how, by who) and
access to population.
• Data analyses superficially described
• Fail to demonstrate expertise in methods, including
technology in using software
• Mixed methods fail to adequately link research aims,
questions, data, analyses
• Unequal attention to qualitative and quantitative
methods when both used
Mark Luborsky MCUAAAR workshop 5/12
26
Imigani y’ikinyarwanda / “hardship and opportunities”
In season, the lake’s bounty for the fish and edges shrink, once hungry crocodiles
enjoy plenty,
until the next rains, when lakes grow and again they are hungry while others have
27
Mark Luborsky MCUAAAR workshop 5/12
plenty