Transcript Slide 1
TM Quantitative and Qualitative Methods for Assessing and Improving Outcomes and Service Quality
Ottawa, Canada June 14, 2006 Presented by: Dr. Colleen Cook, Dean Texas A&M University Project web site – www.arl.org/libqual/
Why Assess?
“ In an
age of accountability
, there is a pressing need for an effective…process to evaluate and compare research libraries.” 700 participants in LibQUAL+ ™ 123 Association of Research Libraries (ARL) alone, over
$3.4 billion dollars
were expended in 2003/2004 Note. M. Kyrillidou and M. Young. (2005).
ARL Statistics 2003-04. Washington, D.C.: ARL, p.5.
Libraries Remain a Credible Resource in 21
st
Century
98% agree with statement , “My … library contains information from credible and known sources.”
Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
Changing Behaviors
Recent Survey:
Only 15.7% agreed with the statement “The Internet has not changed the way I use the library.” Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
Faculty: Dependence on Electronic Resources Will Increase
“I will become increasingly dependent on electronic research resources in the future.” 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% Not Well Somewhat Very Well 2000 2003
http://www.arl.org/arl/proceedings/144/guthrie_files/guthrie.ppt
Research Behavior: Personal Control
When searching for
journals for research:
• • Only 13.9% ask a librarian for assistance
Only 3.2% consider consulting a librarian a preferred way of identifying information
Note. Digital Library Federation and Council on Library and Information Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
Total Circulation
600000 550000 500000 450000 400000 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.
Reference Transactions
170000 160000 150000 140000 130000 120000 110000 100000 90000 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 Note. M. Kyrillidou and M. Young. (2005). ARL Statistics 2003-04. Washington, D.C.: ARL, p.6.
Web Usage
Total File Requests - UT Austin Libraries 2000-2003 900,000,000 800,000,000 700,000,000 600,000,000 500,000,000 400,000,000 300,000,000 200,000,000 100,000,000 0 2000 2001 2002 2003 Total Hits
Enter LibQUAL+ ™
The necessity of assessment Rapid shifts in information-seeking behavior The reallocation of resources from traditional services and functions
The Challenge of Assessment in Libraries
Traditional statistics emphasize inputs, expenditures, acquisitions, holdings, etc.
Helping funding agencies understand success of investment No demonstrable relationship between expenditures and service quality Lack of metrics describing outcomes: success from the user’s point of view Need to redesign library services to better meet changing patterns of use Building the climate, tools, and skill set for library assessment
ARL New Measures Initiative
Collaboration among member leaders with strong interest in this area Specific projects developed with different models for exploration Intent to make resulting tools and methodologies available to full membership and wider community
LibQUAL+ ™ Goals
Improve mechanisms and protocols for evaluating libraries Develop web-based tools for assessing library service quality Identify best practices in providing library service Support libraries seeking to understand changes in user behavior Assist libraries seeking to re-position library services in the new environment
LibQUAL+ ™ Outcomes
Securing information that contributes meaningfully to planning and improvement efforts at a local level Providing analytical frameworks that institutional staff can apply without extensive training or assistance Helping decision-makers understand success of investments Finding useful inter-institutional comparisons
The LibQUAL+ ™ Premise
PERCEPTIONS SERVICE
“….
only customers judge quality; all other judgments are essentially irrelevant”
Note. Zeithaml, Parasuraman, Berry. (1999). Delivering quality service. NY: The Free Press.
13 Libraries English LibQUAL+™ Version 4000 Respondents
PURPOSE Emergent 2000 QUAL
Describe library environment; build theory of library service quality from user perspective
DATA LibQUAL+™ Project
Unstructured interviews at 8 ARL institutions
ANALYSIS
Content analysis: (cards & Atlas TI)
PRODUCT/RESULT QUAN
Test LibQUAL+™ instrument Web-delivered survey Reliability/validity analyses: Cronbachs Alpha, factor analysis, SEM, descriptive statistics Case studies 1 Valid LibQUAL+™ protocol Scalable process Enhanced understanding of user-centered views of service quality in the library environment 2
QUAL
Refine theory of service quality Unstructured interviews at Health Sciences and the Smithsonian libraries Content analysis
QUAL
Refine LibQUAL+™ instrument E-mail to survey administrators Content analysis
QUAN
Test LibQUAL+™ instrument Web-delivered survey Focus groups Reliability/validity analyses including Cronbachs Alpha, factor analysis, SEM, descriptive statistics Content analysis
QUAL
Refine theory
Iterative 2005
700 Libraries English, Dutch, Swedish, German LibQUAL+™ Versions 160,000 anticipated respondents Vignette Re-tooling Cultural perspective 3 Refined survey delivery process and theory of service quality 4 Refined LibQUAL+™ instrument 5 Local contextual understanding of LibQUAL+™ survey responses 6
76 Interviews Conducted
York University University of Arizona Arizona State University of Connecticut University of Houston University of Kansas University of Minnesota University of Pennsylvania University of Washington
Smithsonian Northwestern Medical
LoadedPT:P1:01xxxxxxxxxxxxxx.txt,S:\Admin\Colleen\ServQual Interviews\TEXT Only\01xxxxxxxxx.txt (redirected: c:\zz\atlasti\fred
Dimensions of Library Service Quality
Library Service Quality Affect of Service Empathy Responsiveness Assurance Reliability Library as Place Utilitarian space Symbol Refuge Model 3 Information Control Scope of Content Convenience Ease of Navigation Timeliness Equipment Self-Reliance
Affect of Service
“I want to be treated with respect. I want you to be courteous, to look like you know what you are doing and enjoy what you are doing. … Don’t get into personal conversations when I am at the desk.” Faculty member
Library as Place
“One of the cherished rituals is going up the steps and through the gorgeous doors of the library and heading up to the fifth floor to my study. … I have my books and I have six million volumes downstairs that are readily available to me in an open stack library.” Faculty member
Library as Place
“I guess you’d call them satisfiers. As long as they are not negatives, they won’t be much of a factor. If they are negatives, they are a big factor.” Faculty member
Information Control
“…first of all, I would turn to the best search engines that are out there. That’s not a person so much as an entity. In this sense, librarians are search engines [ just ] with a different interface.” Faculty member
Information Control
“By habit, I usually try to be self-sufficient. And I’ve found that I am actually fairly proficient. I usually find what I’m looking for eventually. So I personally tend to ask a librarian only as a last resort.” Graduate student
Multiple Methods of Listening to Customers
Transactional surveys* Mystery shopping New, declining, and lost-customer surveys Focus group interviews Customer advisory panels Service reviews Customer complaint, comment, and inquiry capture Total market surveys* Employee field reporting Employee surveys Service operating data capture
*A SERVQUAL-type instrument is most suitable for these methods
Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000). Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C
.
LibQUAL+ ™ Resources
An ARL/Texas A&M University joint developmental effort based on SERVQUAL. LibQUAL+™ initially supported by a 3-year grant from the U.S. Department of Education’s Fund for the Improvement of Post-Secondary Education (FIPSE) Initial project established a expert team, re-grounded SERVQUAL concepts, and designed survey methodology Survey conducted at over 500 libraries resulting in a data base of over half a million user responses NSF funded project to refocus LibQUAL+™ on the National Science Digital Library (NSDL)
World LibQUAL+ ™ Survey 2005
Participating Libraries
Rapid Growth in Other Areas
Languages American English British English French Dutch Swedish In development Chinese Greek Spanish German Consortia Each may create 5 local questions to add to their survey Types of Institutions Academic Health Sciences Academic Law Academic Military College or University Community College European Business Hospital Public State Countries U.S., U.K., Canada, the Netherlands, South Africa, Sweden, France, Australia, New Zealand, Malaysia
“22 items”
2000 41-items
Affect of Service Reliability Library as Place
2001 56-items
Affect of Service Library as Place Reliability
2002 25-items
Service Affect
2003 22-items
Service Affect Library as Place Personal Control Library as Place Information Control Provision of Physical Collections Access to Information Self-Reliance Access to Information Information Access
Survey Instrument
“And a Box” Why the Box is so Important
About 40% of participants provide open ended comments, and these are linked to demographics and quantitative data.
Users elaborate the details concerns.
of their Users feel the need to be constructive in their criticisms, and offer specific suggestions for action .
Reliability alpha By Language
By Language Group American (all) British (all) French (all) n 59,318 6,773 172 Service Info.
Affect Lib as Control Place .95
.93
.95
.91
.87
.90
.88
.81
.89
TOTAL .96
.94
.95
Reliability alpha by University Type
By University Type Group Comm Colleges 4 yr Not ARL 4 yr, ARL Acad Health n 4,189 36,430 14,080 3,263 Service Info.
Affect Lib as Control Place .96
.95
.92
.91
.89
.88
.95
.95
.90
.92
.87
.90
TOTAL .97
.96
.96
.96
Validity Correlations
Validity Correlations Serv_Aff Serv_Aff Info_Con 1.0000
.7113
LibPlace TOTALper ESAT_TOT EOUT_TOT .5913
.9061
.7286
.5315
Info_Con .7113
1.0000
.6495
.9029
.6761
.6155
LibPlace .5913
.6495
1.0000
.8053
.5521
.4917
TOTALper .9061
.9029
.8053
1.0000
.7587
.6250
Understanding LibQUAL+ ™ Results
•
Measures the distance between minimally
acceptable
and
desired
service quality ratings
•
Perception ratings ideally fall within the
Zone of Tolerance
Key to Bar Charts
LibQUAL+ ™
2004 Summary Colleges or Universities American English
(n = 69,449)
Score Norms
Norm Conversion Tables facilitate the interpretation of observed scores using norms created for a large and representative sample.
LibQUAL+™ norms have been created at both the individual and institutional level
Institutional Norms for Perceived Means on 25 Core Questions
Note: Thompson, B. LibQUAL+ Spring 2002 Selected Norms, (2002).
LibQUAL+ ™ Interactive Institution Statistics
1)
YEAR
(REQ.) – defaults to current year of the survey; controls the Master List (dynamic) All Master List of Institutions Your List Clear User selects an institution from the Master List, the selection is then added to “Your List”. To avoid duplicate choices, the selection from the Master List will disappear once added to “Your List”.
Other parameters can be added below (institution type, language, consortia, and/ or SAVED LISTS). Text here stating that this section is optional and may be added to “Your List” to narrow down results.
Consortia
, based on current year; dynamic
INSTITUTION TYPE
ADD
LANGUAGE
ADD
CONSORTIA
ADD
Language
(s) will be determined based on selection (s) from the Master List of Institutions; dynamic Text box for user to name and save search parameters for future searches.
SAVED LISTS
ADD 2) • OVERALL – defaults to OVERALL • Dimension (3) 4 items included in this drop-down menu min des per adeq sup
VARIABLES
- Default to perceived SUBMIT SAVE
Results Page
: This page generates a graph, Summary Statistics, Your Statistics, and Norms ( including users’ norm values); restates information entered into the form
1.40
1.20
1.00
0.80
0.60
0.40
0.20
0.00
-0.20
-0.40
Adequacy Gap
The difference between the minimum and perceived score Adequacy Gap
The difference between the minimum and perceived score.
LibQUAL+™ UT Austin ARL Peers
In Closing LibQUAL+ ™
Focuses on success from the users point of view (outcomes) Demonstrates that a web-based survey can handle large numbers; users are willing to fill it out; and survey can be executed quickly with minimal expense Requires limited local survey expertise and resources Analysis available at local and inter-institutional levels Offers many opportunities for using demographics to discern user behaviors
LibQUAL+ ™ Resources
LibQUAL+ ™ Website: http://www.libqual.org
Publications: http://www.libqual.org/publications Events and Training: http://www.libqual.org/events LibQUAL+ ™ Bibliography: http://www.coe.tamu.edu/~bthompson/servqbib LibQUAL+™ Procedures Manual: http://www.libqual.org/Information/Manual/index.cfm
LibQUAL+ ™ Contact Information
MaShana Davis Technical Communications Liaison [email protected]
Richard Groves Customer Relations Coordinator [email protected]
Mary Jackson LibQUAL+™ Services Manager [email protected]
Martha Kyrillidou Director, ARL Statistics and Service Quality Programs [email protected]
woof