Transcript LibQUAL+TM : A New Way of Listening To Users


Service Quality Assessment in a
Digital Library Environment
EDUCAUSE 2002
Duane Webster
Atlanta, Georgia
October 3, 2002
Overview of Discussions
1. New Models for Understanding and
Describing Library Success
2. ARL’s New Measures Initiative
3. LibQUAL+ Project Development
4. Experience with LibQUAL+
5. Conclusions and Next Steps
The Association of Research
Libraries
www.arl.org
Mission:
Shaping and influencing forces affecting the
future of research libraries in the process of
scholarly communication.
Members:
America.
123 major research libraries in North
Ratios:
4% of the higher education institutions
providing 40% of the information resources.
Users:
3 million students and faculty served.
Expenditures: $2.35 billion annually, $727 million for
acquisitions of which 9% is invested in
access
to electronic resources.
ASSOCIATION OF RESEARCH LIBRARIES
ARL New Measures Initiative
 Collaboration among member leaders with
strong interest in this area
 Specific projects developed with different
models for exploration
 Intent to make resulting tools and
methodologies available to full
membership and wider community
LibQUAL+™ Description
LibQUAL+TM is a research and
development project undertaken to define
and measure library service quality across
institutions and to create useful qualityassessment tools for local planning.
Project Resources
LibQUAL+TM is an ARL/Texas A&M
University joint effort. The project is
supported in part by a 3-year grant from the
U.S. Department of Education’s Fund for the
Improvement of Post-Secondary Education
(FIPSE) and a 3-year grant from the
National Science Foundation (NSF)
The Imperative
for our Research
In an age of accountability, there is a
pressing need for an effective and practical
process to evaluate and compare research
libraries. In the aggregate, among the 122
Association of Research Libraries (ARL)
alone, over $2.8 billion dollars were
expended in 1999/2000 to satisfy the library
and information needs of the research
constituencies in North America (Kyrillidou
& Young, 2001, p. 5).
The Problem of Assessment in
Research Libraries
 The lack of metrics
 ARL Membership Criteria Index variables
emphasize inputs, primarily expenditures
 No demonstrable relationship between
expenditures and service quality
 To rise in the ARL Index it is only
necessary to spend more
Assessment
“The difficulty lies in trying to find a single model or
set of simple indicators that can be used by
different institutions, and that will compare
something across large groups that is by definition
only locally applicable—i.e., how well a library
meets the needs of its institution. Librarians have
either made do with oversimplified national data or
have undertaken customized local evaluations of
effectiveness, but there has not been devised an
effective way to link the two.” Sarah Pritchard,
Library Trends, 1996
LibQUAL+™ Project Goals
 Establishment of a library service quality
assessment program at ARL
 Development of web-based tools for
assessing library service quality
 Development of mechanisms and
protocols for evaluating libraries
 Identification of best practices in
providing library service
LibQUAL+TM Participants
Year 3
Year 2
164 Participants
43 Participants
Year 1
12 Participants
Spring 2000
Spring 2001
For More Information about Participants:
Visit the LibQUAL+ web site.
Spring 2002
Relationships: perceptions,
service quality and satisfaction
….only customers judge quality;
all other judgments are essentially
irrelevant”
Zeithaml, Parasuraman, Berry. (1999).
Delivering quality service. NY: The Free Press.
Source: Parasuraman, ARL Symposium on Measuring Service Quality
Washington, DC, October 2000
70+ Interviews conducted
 York University
 University of
 University of
Minnesota
 University of
Pennsylvania
 University of
Washington
 Smithsonian
 Northwestern
Medical
Arizona
 University of
Connecticut
 University of
Houston
 University of Kansas

Atlas TI Dimensions of Analysis
Library
Service
Quality
Affect of Service
Reliability
Empathy
Responsiveness
Comprehensive
Collections
Assurance
Ubiquity and Ease
of Access
Self-reliance
?
Library as Place
Formats
Utilitarian space
Timely access to
resources
Symbol
Physical location
Refuge
Figure 1: Dimensions of Library Service Quality
Self-reliance
“By habit, I usually try to be self-sufficient. And
I’ve found that I am actually fairly proficient. I
usually find what I’m looking for eventually.
So I personally tend to ask a librarian only as a
last resort.”
Graduate student
Self-reliance
“…first of all, I would turn to the best search
engines that are out there. That’s not a person so
much as an entity. In this sense, librarians are
search engines [ just ] with a different interface.”
Faculty member
Affect of Service
“I want to be treated with respect. I want you to
be courteous, to look like you know what you
are doing and enjoy what you are doing. …
Don’t get into personal conversations when I
am at the desk.”
Faculty member
Comprehensive Collections
“I think one of the things I love about academic
life in the United States is that as a culture…,
we tend to appreciate the extraordinary
importance of libraries in the life of the mind.”
Faculty member
Comprehensive Collections
“I sense that if I were in an institution that didn’t
have the rich collections as this library and the
very effective staff members that this library
has that I would imperceptibly slip in my
discipline….”
Faculty member
Ubiquity of Access
“Over time my own library use has become
increasingly electronic. So that the amount of
time I actually spend in the library is getting
smaller and the amount of time I spend at my
desk on the web … is increasing.”
Faculty member
Reliability
“You put a search on a book and it’s just gone;
it’s not reacquired. … There’s more of a
problem of lost books, of books that are gone
and nobody knows why and nobody’s doing
anything about it.”
Faculty member
Reliability
“I put something on reserve. And it didn’t show
up, and somebody complained. I went back
and said ‘I’ve asked for this to be put on
reserve’ and they had lost the form. So I had
to do it again.”
Faculty member
Library as Place
“I guess you’d call them satisfiers. As long as
they are not negatives, they won’t be much
of a factor. If they are negatives, they are a
big factor.”
Faculty member
Library as Place
“The poorer your situation, the more you need the
public spaces to work in. When I was an
undergraduate, I spent most of my time in the
library, just using it as a study space.”
Faculty member
Library as Place
“One of the cherished rituals is going up the steps
and through the gorgeous doors of the library
and heading up to the fifth floor to my study. …
I have my books and I have six million volumes
downstairs that are readily available to me in an
open stack library.”
Faculty member
Dimensions of Library
Service Quality
Affect of Service
Library
Service
Quality
Personal Control
Empathy
Ease of Navigation
Responsiveness
Convenience
Assurance
Modern Equipment
Reliability
Information Access
Library as Place
Scope
Utilitarian space
Timeliness
Symbol
Convenience
Refuge
Affect of Service
 Emerged as the dominant factor early in our
work
 Absorbed several of the original
SERVQUAL questions measuring
Responsiveness, Assurance and Empathy
 In the current analysis also includes
Reliability
 All in all: the Human Dimension of Service
Quality
Library as Place
 Transcends the SERVQUAL dimension of
Tangibles to include the idea of the library as
the campus center of intellectual activity
 As long as physical facilities are adequate,
library as place may not be an issue
Personal Control
 How users want to interact with the
modern library
 Personal control of the information
universe in general and web navigation
in particular
Access to Information
 Ubiquity of access: information delivered
in the format, location and time of choice
 Comprehensive collections
LibQUAL+ 2002 Iteration
 42 — ARL Libraries
 35 — Health Sciences Libraries
 36 — State Colleges & Universities
(excluding ARL)
 34 — Private Colleges & Universities
(excluding ARL)
 15 — Community Colleges
 2 — Special & Public Libraries
(Smithsonian & NYPL)
The Challenge of Analysis
 There are few, if any useful conclusions to be
drawn from aggregate data of all institutions,
because their missions and subsequent user
expectations for service are too diverse.
 There are commonalities in service delivery
profiles that merit further investigation.
 In the long run, information that may be
derived from demographic responses of
individuals may yield the richest data.
Two Interpretation
Frameworks
 Score Norms
 Zone of Tolerance
Zone of Tolerance
 The area between minimally acceptable
and desired service quality ratings
 Perception ratings ideally fall within the
Zone of Tolerance
Aggregate Dimension
Summary (n=70,445)
Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 1, p. 24
Faculty Item Summary
Note: LibQUAL+ Spring 2002 Aggregate Survey Results. (2002). vol. 2, p. 40
Score Norms
 Norm Conversion Tables facilitate the
interpretation of observed scores using norms
created for a large and representative sample.
 LibQUAL+TM norms have been created at
both the individual and institutional level
Overall Mean Scores and Service Adequacy Gap Scores
By Cohort Group
2002 LibQUAL+ Iteration
(n=162)
Community
Colleges
7.26
(.55)
Private
Colleges
6.90
(.49)
AAHSL
7.07
(.56)
State Colleges
& Universities
6.38
(.30)
ARL
Top 40
6.84
(.46)
ARL
Other
6.74
(.27)
LibQUAL+™ Fundamental Contributions
to the Measurement of Effective Delivery
of Library Services
 Shift the focus of assessment
from mechanical
expenditure-driven metrics to
user-centered measures of
quality
 Determine the degree to which
information derived from local
data can be generalized,
providing much needed “best
practices” information
 Re-ground gap theory for the
library sector, especially
academic libraries
 Demonstrate the efficacy of
large-scale administration of
user-centered assessment
transparently across the web
 Grounded questions yield
data of sufficient granularity
to be of value at the local
level
 Makes little demand of local
resources and expertise
Recognize the limitations of
listening to customers
 Customers have a limited frame of
reference and tend to offer incremental,
rather than bold, suggestions
– A better slide rule
– The microwave oven, Post-it Notes,
Velcro
 Innovation is the responsibility of staff
Anthony W. Ulwick, Harvard Business Review, January 2002
Shift the focus to outcomes
 Plan outcome-based customer interviews
 Capture desired outcomes
 Organize the outcomes
 Rate the outcomes for importance and satisfaction
– Opportunity algorithm:
(Importance+(Importance-Satisfaction)=Opportunity)
 Use the outcomes to jump-start innovation
Anthony W. Ulwick, Harvard Business Review, January 2002
When desired outcomes
become the focus of customer
research, innovation becomes
a manageable, predictable
discipline.
Anthony W. Ulwick, Harvard Business Review, January 2002
Summary
 Survey can handle large numbers
 Survey can be turned around quickly
 Limited local expertise required
 Interpretations should be across chosen
cohorts
 Lots of opportunities for using demographics
to discern user behaviors
 Q-technique and other tests will provide
opportunities to observe how institutions
may cluster
LibQUAL+ Related
Documents
LibQUAL+ Web Site
http://www.arl.org/libqual/
LibQUAL+ Bibliography
http://www.coe.tamu.edu/~bthompson/servqbib
Survey Participants Procedures Manual
http://www.arl.org/libqual/procedure/lqmanual2.pdf
NDSL LibQUAL+ Activities
• 120-200 qualitative interviews to contribute
to identifying dimensions of digital library
service quality
• Test and refine dimensions of digital library
service quality and self-sufficiency through
development of total market survey
• Implement survey across variety of
organizational and digital library
implementations
NDSL LibQUAL+ Goals
• Define dimensions of digital library service
quality from the users’ perspectives
• Develop tool for measuring user perceptions
and expectations of digital library service
quality across NSDL digital library contexts
• Identify digital library best practices that
permit generalizations across operations and
development platforms
NSF Grant
• Assess service quality in digital
libraries
• 3 year period
• Adopt LibQUAL+ instrument for use
in the Science, Math, Engineering and
Technology Education Digital Library
community (NSDL)
LibQUAL+TM Project History
9/99 - ARL launches “New Measures Initiative” which includes the study of service
effectiveness known as SERVQUAL spearheaded by Texas A&M University.
1/00 - Initial 12 institutions begin the SERVQUAL study led by the Texas A&M team.
7/00 - LibQUAL+ as a distinct library-based assessment tool is presented.
9/00 - ARL and Texas A&M awarded a FIPSE grant to fund further development of the
LibQUAL+ project.
10/00 - The ARL symposium, “New Culture of Assessment in Academic Libraries Measuring
Service Quality” attracts a group of 170 people.
1/01 - Representatives from 43 research and university libraries participating in the Spring 2001
Implementation meet in Washington, DC during ALA midwinter.
6/01 - National Science Foundation awards grant to ARL and Texas A&M to adapt LibQUAL+
for NSDL
TM
LibQUAL+ Team
•
•
•
•
•
•
ARL
Duane Webster
Martha Kyrillidou
Kaylyn Hipps
Julia Blixrud
Jonathan Sousa
Consuella Waller
•
•
•
•
•
•
TAMU
Fred Heath
Colleen Cook
Bruce Thompson
Yvonna Lincoln
Trey Thompson
Julie Guidry
Overview of Discussions
1. New Models for Understanding and
Describing Library Success
2. ARL’s New Measures Initiative
3. LibQUAL+ Project Development
4. Experience with LibQUAL+
5. Conclusions and Next Steps
The Problem of Assessment in
Research Libraries
• ARL Membership Criteria Index variables
emphasize inputs, primarily expenditures
• To rise in the ARL Index it is only
necessary to spend more
• No demonstrable relationship between
expenditures and service quality
• The lack of metrics describing
performance