Transcript Power Analysis: NIH Applications for Research Funding

A Statistical Perspective on
Mixed Methods
MIXED METHODS RESEARCH WORKSHOP
Institute For Social Research – Room 6080
M a y 1 6 th, 2 0 1 2
Thomas N. Templin, PhD
Office of Health Research
College of Nursing
Wayne State University
Please do not duplicate or use these slides without the express permission of the author.
Research designs in context
 The literature on mixed methods (MM) is large and
complex so it is worth taking a look first at what we
stand to gain.

Put these developments in the context of research design
today.
 The growth of MM designs over the past few decades
is part of growing trend to make research designs
more intelligent, more capable, and more adaptive.
Tom Templin, MCUAAAR workshop 5/2012
Research designs in context
 Intelligent
 Uses what we know to learn what we don’t know
 Example: pretest information
 Capable
 Arrives at the correct population inference
 Example: all statistical assumptions are met
 Adaptive
 Does not waste resources
 Example: stopping early
Tom Templin, MCUAAAR workshop 5/2012
Research designs in context
 Experimental designs and clinical trials
 Interim analyses allow multiple looks at the data without
inflating Type 1 error.

Trial can be stopped early for either success or failure (futility).

Sequential multiple assignment randomized trial (SMART) for
the development of dynamic treatment regimes. Susan
Murphy

Brown, C. H., T. R. Ten Have, et al. (2009). Adaptive Designs
for Randomized Trials in Public Health. Annual Review of
Public Health. 30: 1-25.
Tom Templin, MCUAAAR workshop 5/2012
Research designs in context
 Quasi experimental designs extended the
range the interventions we could examine
 Cook,
T. D. and D. T. Campbell (1979). Quasi-Experimentation:
Design and Analysis Issues for Field Settings. Chicago, Rand
McNally.
 Shadish,
W. R., Cook, T. D., & Campbell, D. T. (2002).
Experimental and Quasi-Experimental Designs: for
Generalized Causal Inference.
Tom Templin, MCUAAAR workshop 5/2012
Research designs in context
 Causal Modeling


Development of causal modeling theory increased the
sophistication of research questions we could address
including threats to validity in randomized designs and
quasi-experiments

Rubin, D. B. (1974). Estimating Causal Effects of Treatments in Randomized and Nonrandomized
Studies. Journal of Educational Psychology, 66(5).

Pearl, J. (2000). Causality: Models, reasoning, and inference. New York, NY, Cambridge University
Press.

Morgan, S. L., & Winship, C. (2007). Counterfactuals and Causal Inference: Methods and Principles
for Social Research. New York: Cambridge Univ Press.

Hernan, M., & Robins, J. (2012). Causal Inference: Chapman & Hall/CRC.
Equations plus qualitative information
Tom Templin, MCUAAAR workshop 5/2012
Research designs in context
 The Hope for Mixed Methods




Increased credibility of qualitative findings
Increased role of qualitative exploration to shape the
direction of the research
Increased value of individual differences and
individual variation
More intelligent, capable, adaptive
 Jennifer
Green’s mixed method story
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
 We have routinely used open coded items on surveys
in convergent (concurrent) designs.
 We have always used multi-phased sequential
designs in instrument development.
 It is not uncommon to include qualitative interviews
to determine how participants experience an
intervention in embedded designs
 We just didn’t use these particular terms
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
 We have routinely used open coded items in survey
designs.
 We have always used multi-phased designs in
instrument development.
 It is not uncommon to include qualitative interviews
to determine how participants experience an
intervention
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
 Mixed methods research design terminology
 convergent (concurrent or parallel)
 Sequential (exploratory sequential, explanatory sequential)
 Embedded
 Terminology to describe how qualitative and
quantitative data can be integrated



Merged
Connected
Embedded
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
 NIH OBSSR “Best Practices for Mixed Methods
Research in the Health Sciences

Directions for mixed methods research applications to NIH
 Explicit reference to qualitative and quantitative data
in the Specific Aims, Significance and Innovation
sections
 Description of rigorous qualitative and quantitative
methodology in the Approach Section
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
In a concurrent design, investigators often discuss the collection
of both types of data (quantitative and qualitative) before
discussing the analysis of both types of data (quantitative and
qualitative).
Data Collection
Quantitative
Qualitative
Data Analysis and Interpretation
Quantitative
Qualitative
Integration/Merging Procedures
Tom Templin, MCUAAAR workshop 5/2012
What’s New in Mixed Methods?
In a sequential design discuss the collection and analysis of the
first type of data (quantitative or qualitative) and then discuss the
collection and analysis of the subsequent type of data (qualitative
or quantitative).
First Phase (quantitative or qualitative)
Data Collection
Data Analysis and Interpretation
Connecting Procedures (e.g., development of sampling procedures
or materials based on the results from the first phase)
Second Phase (qualitative or quantitative)
* Data Collection
* Data Analysis and Interpretation
Tom Templin, MCUAAAR workshop 5/2012
Three Ways of Integrating
Qualitative and Quantitative Data
MERGING
CONNECTING
EMBEDDING
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
Merging—1. Qualitative and Quantitative data can be
integrated by using words and numbers together.
Describing the statistical result and giving examples
from the qualitative analysis. In the opening Chapter
of Creswell and Plano Clark’s (2011) handbook on
mixed methods they give this description of mixed
methods as used in everyday life.
Creswell, J. W. & Plano Clark (2011) Designing and Conducting Mixed
Methods Research. Thousand Oaks, CA: Sage Publications Inc.
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
Merging qual and quant data Example 1.
“Consider for a moment An Inconvenient Truth, the awardwinning documentary on global warming featuring the
former U.S. vice president and Nobel Prize winner Al Gore ..
In the documentary, Gore narrated both the statistical trends
and the stories of his personal journey related to the changing
climate and global warming. This documentary brings
together both quantitative and qualitative data to tell the
story. Also, listen closely to CNN's broadcast reports about
hurricanes or about the votes cast in elections. The trends are
again supported by the individual stories” (Creswell & Plano
Clark, 2011).
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
 Merging ql and qn data-Example 2.
 Numerically coded qualitative data can be merged with
quantitative data in cross tabulated tables. For example if a
qualitative analysis classifies participants in three mutually
exclusive classes, say Type A, Type B, and Type C. Then it
might be in interest to examine the Type by Gender
crosstab.
 This crosstab is an example of integration by merging
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
 Merging ql and qn data-Example 3.
 If the survey contained both Likert type response items and
qualitative items with open codes items, it might be of
interest to examine the correlation between the Likert items
and various recodings of the open ended item.
 A specific example is actual report card grade and self
reported grade in response to the question, What kinds of
grades do you generally get in school?
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
 Merging ql and qn data-Example 4.
 Short answer questions added to standardized test are
coded and factor analyzed with other test items
Tom Templin, MCUAAAR workshop 5/2012
Integrating qualitative and quantitative
data by merging
 Mixed method research designs that collect qual and quant
data concurrently combine qual and quant by merging as
described in the examples above.
Tom Templin, MCUAAAR workshop 5/2012
Integrating qual and quant data by
connecting.
 Data are connected when the results of the qual (or
quant) in one phase of the study depend in some way
on the quant (or qual) data collected in another
phase of the study.
 Connecting ql and qn--Example 1
Focus groups and experts are used during Phase 1 to
develop questionnaire items for a test that will be
administered to the target population in Phase 2.
Tom Templin, MCUAAAR workshop 5/2012
Integrating qual and quant data by
connecting.
 Connecting ql and qn--Example 2
 In Phase 1, the quantitative analysis ranks students in
terms of their reading ability. In Phase 2, the top 5 and the
lowest 5 students are selected of in depth interviews about
reading attitudes, interests, etc.
 Mixed method sequential research designs use
connecting to integrate qual and quant data.
Tom Templin, MCUAAAR workshop 5/2012
Integrating qual and quant by embedding
 Embedding ql and qn—Example 1.
 “In this form of integration, a dataset of secondary
priority is embedded within a larger, primary design.
An example is the collection of supplemental
qualitative data about how participants are
experiencing an intervention during an experimental
trial” (p. 6).
Tom Templin, MCUAAAR workshop 5/2012
Integrating qual and quant by embedding
 Embedding ql and qn—Example 2.
 “The [researchers] implemented an RCT study to
compare the two treatments in terms of various
repeated measure patient outcomes, including pain
levels. Embedded within the RCT study, they also
gathered qualitative data in the form of audiotapes of
the intervention sessions, along with nurse and
patient notes, to describe the issues, strategies, and
interactions experienced during the intervention.
The results provide evaluation of both the outcomes
and process of the intervention.” (BPMM, p. 6).
Tom Templin, MCUAAAR workshop 5/2012
Designs for Mixed Methods
Research
CONVERGENT
SEQUENTIAL
EMBEDDED
Tom Templin, MCUAAAR workshop 5/2012
The OBSSR Best Practices document gives examples of three
kinds of mixed methods research designs and notes that many
more are available in the literature.
 That is an understatement. Interest in these methods has resulted in a
plethora of MM research designs and frameworks for research designs
since the Jennifer Greene’s (1989) pioneering review. This can make
literature difficult for the novice.



Creswell and Plano Clark (2011) listed 15 different
classification schemes. Most used different terminology.
Onwueegbuzie and Collins (2007) noted that a popular
handbook of mixed methods contained approximately 35 MM
designs.
Jennifer Greene (2007) stated from Maxwell and Loomis, that
“the actual diversity in mixed methods studies is far greater
than any typology can adequately encompass” (2003), pl. 244).
Tom Templin, MCUAAAR workshop 5/2012
Some discontent surrounding mixed method
design typologies
 In Greene’s mixed method text (2007) she writes, “Muted by the
emphasis on design typologies are possible contributions to better
understanding that could come from mixes in philosophy, substantive
theory, and disciplinary thinking, alongside mixes of differences in
personal experience, education, values, and beliefs” (p 15).
 Katrin Niglas (2009) noted that in their, “attempts to build more and
more exhaustive typologies” … “the terminology in the field: it gets very
specific and complicated but remains ambiguous at the same time.
 She attempted to deconstruct MM design categories using conventional
research design concepts. An interesting approach.
Niglas, Katrin (2009). How the novice researcher can make sense of mixed methods designs.
International Journal of Multiple Research Approaches, 1, 13 - 33.
Tom Templin, MCUAAAR workshop 5/2012
Convergent (or parallel or concurrent)
MM design.
 Example from Tony Omwuegbuzie (2007). A study
interested in student attitudes toward reading and
reading strategies administered a survey containing
predetermined questions with Likert type response
options and open ended questions to elicit
qualitative information about reading strategies.
 Since all the information is collected at one time,
standard research design terminology would call this
a cross sectional design.
Tom Templin, MCUAAAR workshop 5/2012
Sequential MM design (or explanatory
sequential or exploratory sequential)
 One data set builds on another. This design involves
two distinct interactive phases. Data is collected and
analyzed in each phase. The results of Phase 1 inform
data collection in Phase 2.
 Example 1. Interviews are conducted following the
administration of a quality of life instrument to
better understand the mechanisms underlying the
responses.
 What else do we need to specify here?
Tom Templin, MCUAAAR workshop 5/2012
Sequential MM design (or explanatory
sequential or exploratory sequential)
 Example 2. Qualitative interviews are conducted to
identify risks involved in the treatment of diabetes. A
questionnaire is developed and administered to a
population of patients.
 Typical instrument development design (exploratory
sequential); type of data integration is connecting.
 See example on connecting.
Tom Templin, MCUAAAR workshop 5/2012
Embedded MM design
 The embedded design can be a variation of a
convergent or sequential design but either qual or
quant is dominant throughout.
 “A prototype would be to conduct an intervention
study and to embed qualitative data within the
intervention procedures to understand how
experimental participants experience the treatment.”
Tom Templin, MCUAAAR workshop 5/2012
Questions
 Seems like the convergent design is the same as a
cross sectional design in monomethod research and
the sequential design is a phased design approach.
Tom Templin, MCUAAAR workshop 5/2012
Conclusions
MIXED METHODS IN CONTEXT
NEW TERMINOLOGY
NEW BEST PRACTICE GUIDELINES
Tom Templin, MCUAAAR workshop 5/2012
Mixed methods in context
 Think about using mixed methods with advances
used in mono method designs


Adaptive experimental designs
Causal modeling
Tom Templin, MCUAAAR workshop 5/2012
New terminology
 Terms advanced in the Best Practice Guidelines are
probably worth knowing even if not intuitive

Data integration terms
Merging
 Connecting
 Embedding


Research design
Convergent (concurrent or parallel)
 Sequential (sequential exploratory and sequential explanatory)
 Embeded

Tom Templin, MCUAAAR workshop 5/2012
New best practice guidelines
 Guidelines advocate discussion of qual and quant in
each part of the application
 While qual methods are often used in designs that
are not mixed methods designs, mixed methods
greatly extends the role of qualitative data by
including qual research in study aims, significance,
and innovation
Tom Templin, MCUAAAR workshop 5/2012
Please do not duplicate or use these slides without the express permission of the author.