Transcript Qualitative Assessment in Libraries: Adding Meaning to

Qualitative Research
Methods for Assessing
Services and Programs in
Academic Libraries
Dr. Robert V. Labaree
Applied Social Sciences Librarian
University of Southern California
[email protected]
CARL 12th Biennial Conference
Hyatt Regency, Irvine, California
April 2-5, 2008
Introduction
Topics to be covered:
 Methodological features of qualitative studies
 When to apply qualitative methods in assessing
library services and programs
 Some techniques for gathering data in library settings
 Benefits and pitfalls associated with qualitative
research
 Recommended resources
 Conclusion and contact information
Slide 2 of 20
General Design Characteristics of
Qualitative Research

Naturalistic
Investigate social processes as they unfold naturally rather than
reasoning from either the conditions under which they occur or
the outcomes that correlate with them.

Emergent
Willingness to adapt methods of analysis as knowledge and
understanding deepens

Purposeful
Cases are selected because they are “information rich”
Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3rd ed. (Thousand Oaks, CA: Sage, 2002), pp. 4041. [and next two slides]
Slide 3 of 20
Data Collection Characteristics of
Qualitative Research

“Thick description”
The goal is deep understanding about phenomena and
experiences

Personal experience and engagement
The researcher is the primary instrument of data collection and
analysis

Empathic stance
Sensitivity, respect, awareness, responsiveness, and openness

Attention to process
Situational and contextual changes occur throughout the life of
the study
Slide 4 of 20
Analytical Characteristics of
Qualitative Research

Unique case orientation
Every case is special and unique

Inductive and creative
Research process guided by analytical principles rather than rules

Holistic perspective
Phenomena under study are complex and cannot be reduced to a few variables

Context sensitivity
Avoid broad generalizations; investigation does not rest in isolation from the
larger social/cultural/economic/political environment

Reflexivity
Critical self-awareness of one’s own presence in the research process
Slide 5 of 20
Selective Applications in
Library Settings









To assess individualized outcomes
To explore internal dynamics of programs and services
To evaluate critical cases of program or service utilization
To acquire details about program implementation
To obtain information about the nuances of program or
service quality
To personalize the evaluation process and/or outcomes
assessment
To discover effects of specific programs or services
To add depth, detail, and meaning to statistical data
To provide privileged access to processes, causes, and
effects.
Source: Patton, Michael Q. How to Use Qualitative Methods in Evaluation. 1st ed. (Thousand Oaks, CA: Sage, 1987), pp. 41-42.
Slide 6 of 20
Some Techniques for Collecting
Assessment Data
In-Depth Interviews – speaking with and listening to
our users
Observation – witness what is going on in a particular
setting and explore the experiences of others
Qualitative Case Studies – holistic understanding of
social phenomena in library settings
Slide 7 of 20
In-Depth Interviews
Purpose
In-depth interviewing seeks to achieve the same level
of knowledge and understanding possessed by the
respondent and to understand personal
experiences and perceptions within a
contextualized, social framework.
Johnson, John M. “In-Depth Interviewing.” In Handbook of Interview Research: Context and Method.
Jaber F. Gubrium and James A. Holstein, eds. (Thousand Oaks, CA: Sage), pp. 103-119.
8 of 20
In-Depth Interviews Design
Considerations
General Typologies
Structured, semi-structured, conversational
Seven Stages of an Interview Investigation (Kvale, 1996)
1. Thematizing—constructing the study’s purpose
2. Designing—protocol of how interview is to be undertaken
3. Interviewing—reflexive and ethical
4. Transcribing—preparing data for analysis
5. Analyzing—coding the data and discovering themes
6. Verifying—ascertain validity and reliability
7. Reporting—communicate findings to stakeholders
9 of 20
In-Depth Interviews
Outcomes for Assessment





What is meaningful or important is in the
respondent’s own voice
Researcher’s knowledge, expertise, and interpersonal
skills can be used to explore interesting or unexpected
concepts or issues raised by respondents
Opportunity for deep probing of issues
May reveal previously unknown personal issues or
experiences
Quotes have impact!
Source: Patton, Michael Q. Qualitative Research and Evaluation Methods. 3rd ed. (Thousand Oaks, CA:
Sage, 2002)
10 of 20
Observation
Purpose
A method enabling researchers to systematically
learn about and document the activities of
respondents in their natural, organizational
setting by observing and/or participating in those
activities.
Kawulich, Barbara B. “Participant Observation as a Data Collection Method.” Forum: Qualitative Social Research 6 (May
2005): Article 43 [http://www.qualitative-research.net/fqs-texte/2-05/05-2-43-e.htm]; Lofland, John and Lyn H.
Lofland. Analyzing Social Settings: A Guide to Qualitative Observation and Analysis. 3rd ed. Belmont, CA:
Wadsworth, 1995.
11 of 20
Observation Design Considerations
General Forms
 Participatory Observation
 Passive Observation of Interactions
 Covert Observation of Interactions
 Inobtrusive Observation concerning the Residues and
Consequences of Interactions
Observation is useful:
 When you need direct information about phenomena
 When you are attempting to understand an ongoing behavior,
process, event, or situation as it unfolds in real time
 When there is physical evidence or outcomes that can be
readily seen
 When written or discursive forms of data collection procedures
appear to be inadequate or inappropriate
12 of 20
Observation
Outcomes for Assessment

Can facilitate multidimensional and long-term
imbeddedness in the “natural setting”
 Melds together aspects of looking and listening and
watching and asking (connect what people are saying
to what they are actually doing)
 Unmasks everyday interaction between people and
organizations; reveals the mundane
 Cases can be revisited repeatedly
13 of 20
Qualitative Case Studies
Purpose
May be considered a unit of analysis or a choice of
what is to be studied; the case is a single bounded
system that can encompass time (development or
history of a program), space (the object of study is
located in a particular place), and/or groupings of
components (number of participants; types of
programs).
Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: Sage, 1995; Case Study Evaluations.
Washington, DC: United States General Accounting Office, Program Evaluation and Methodology Division, 1990.
14 of 20
Case Studies Design Considerations
Cases can be:

Individuals—comparative sample of individual experiences
 Groups—sample of individuals who share a common identity
 Communities—sample of individuals with ethnic, cultural,
spiritual or other types of characteristics in common
 Programs—comparative sample of projects or services
intended to meet a public need
 Policies—comparative sample of plans of action
 Organizations—comparative sample of a formal group of
people with one or more shared goals
 Events—sample of noteworthy occurrences that trigger an
organizational response or state of transition
15 of 20
Qualitative Case Studies
Applications for Assessment






Illustrative: Descriptive and intended to add “realism” and in-depth
examples to other information about a program or policy.
Exploratory: Also descriptive, but aimed at generating hypotheses for later
investigation rather than for purposes of illustrating.
Deviant/Critical Instance: Examines a single instance of unique interest
or serves as a critical test of an assertion about a program, problem, or
strategy.
Program Implementation: Investigates operations, often at several sites,
and with a subjective, value laden, and/or affective emphasis.
Program Effects: This application uses the case study to examine causality
and usually involves multi-site, multi-method assessments.
Cumulative: This brings together findings from many case studies to
answer an evaluation question, whether descriptive, normative, or causeand-effect.
16 of 20
Other Techniques for Assessment
Student Portfolios
A form of individual performance assessment “in which students’ work is
systematically collected and carefully reviewed for evidence of learning
and development” [Palomba and Banta, 1999, p. 131].
Content Analysis
A set of techniques used for the systematic analysis of communicative content
[e.g., text, discourse, images, etc.]
Life Histories
A practice of collecting oral testimonies intended to “advance understanding
about the complex interactions between individuals’ lives and the
institutional and societal contexts in which they are lived” [Cole and
Knowles, 2001, p. 126]
17 of 20
Some Benefits
Some Pitfalls


Provides deep meaning
 Brings contextual clarity to
lived experiences
 Enhances subjective
awareness of others
 Reveals the hidden or
mundane
 Aims are not pre-determined
 Provides a multidimensional and dynamic
picture of group experience
 Helps determine questions
and types of follow-up
research
Institutional Review Boards
 Quantity of information
 Time and energy
 Decision-makers unfamiliar
with qualitative methods
and their application to
problem-solving
 Ethical issues
 Difficult to separate
phenomena into distinct and
workable units of analysis
 Inquiries do not lend
themselves to replicability
or generalizability
Selected Resources








Creswell, John W. Qualitative Inquiry and Research Design: Choosing Among
Five Traditions. Thousand Oaks, CA: Sage Publications, 1998.
Denzin, Norman K. and Yvonna S. Lincoln, eds. Handbook of Qualitative
Research. 1st edition. Thousand Oaks, CA: Sage Publications, 1994.
Hatch, J. Amos. Doing Qualitative Research in Educational Settings. Albany,
NY: State University of New York Press, 2002.
Kvale, Steiner. InterViews: An Introduction to Qualitative Research
Interviews. Thousand Oaks, CA: Sage Publications, 1996.
Lofland, John and Lyn H. Lofland. Analyzing Social Settings: A Guide to
Qualitative Observation and Analysis. Belmont, CA: Wadsworth, 1995.
Patton, Michael Q. Qualitative Research and Evaluation Methods. 3rd edition.
Thousand Oaks, CA: Sage Publications, 2002.
Stake, Robert E. The Art of Case Study Research. Thousand Oaks, CA: Sage
Publications, 1995
Stern, Elliot, ed. Evaluation Research Methods. 4 vols. Thousand Oaks, CA:
Sage Publications, 2005.
Contact Information
Dr. Robert V. Labaree
Applied Social Sciences Librarian
Von KleinSmid Center Library
University of Southern California
Los Angeles, CA 90089-0048
213-740-5824
[email protected]
20 of 20