Transcript Where's the Evidence

Old and New Measures …
Why Bother?
Georgetown University Library
February 24, 2003 • Washington, DC
Martha Kyrillidou
Senior Program Officer for Statistics and Measurement
Consuella Askew
LibQUAL+™ Program Officer
Association of Research Libraries
www.arl.org
Association of Research Libraries
• Mission: Shaping and influencing factors affecting
the future of Research Libraries in the process of
scholarly communication
• Members: 124 major research libraries in North
America
• Users: 3 million students and faculty served
• Service measures: 73 million circulations
18.5 million reference transactions
108 thousand group training lessons
1.6 million participants
Association of Research Libraries
www.arl.org
ARL Strategic Objectives
• 8 including federal relations, scholarly
communication, preservation, collections … and
•PERFORMANCE MEASURES
To describe and measure the performance
of research libraries and their contribution
to teaching, research, learning and
community service
Association of Research Libraries
www.arl.org
ARL Statistics and Trends
• ARL Statistics:
Serial and Monograph Costs
Interlibrary Loan/Document Delivery
Expenditures in ARL Libraries
Expenditures for Electronic Resources
•ARL Annual Salary Survey
Salaries and Human Resources Issues
• Preservation Statistics
• ARL Law and Medical Library Statistics
• LibQUAL+
• E-Metrics
Association of Research Libraries
www.arl.org
Graph 4
Expenditure Trends
in ARL Libraries, 1986-2001
Serial Expenditures
(+210%)
200%
Library Materials
(+172%)
% Change Since 1986
170%
140%
TOTAL (+113%)
Operating
Expenditures
(+102%)
110%
Total Salaries
(+100%)
Monograph
Expenditures (+66%)
80%
50%
CPI
(+62%)
20%
-10%
1986
1988
1990
So u r c e :
1992
1994
1996
A R L St a t i s t i c s 2 0 0 0 - 0 1 , A s s o c i a t i o n o f R e s e a r c h L i b r a r i e s , W a s h i n g t o n , D . C .
Association of Research Libraries
www.arl.org
1998
2000
Graph 2
Monograph and Serial Costs
in ARL Libraries, 1986-2001
250%
Serial Unit Cost
(+215%)
Serial Expenditures
(+210%)
% Change Since 1986
200%
150%
100%
Monograph Unit Cost
(+68%)
50%
Monograph
Expenditures
(+66%)
Serials Purchased
(-5%)
0%
-50%
1986
Monographs
Purchased
(-9%)
1988
1990
So u r c e :
1992
1994
1996
A R L St a t i s t i c s 2 0 0 0 - 0 1 , A s s o c i a t i o n o f R e s e a r c h L i b r a r i e s , W a s h i n g t o n , D . C .
Association of Research Libraries
www.arl.org
1998
2000
Graph 3
Supply and Demand
in ARL Libraries, 1986-2001
Interlibrary
Borrowing
(+206%)
% Change Since 1986
200%
150%
100%
50%
0%
-50%
1986
1988
1990
1992
1994
1996
1998
2000
Source: ARL Statistics 2000-01 , Association of Research Libraries, Washington, D.C.
Association of Research Libraries
www.arl.org
Interlibrary
Lending
Grad. Students
(+81%)
(+39%)
Faculty (+15%)
Total
Students (+14%)
Serials
Purchased (-5%)
Monographs
Purchased
(-9%)
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Georgetown Libraries, 2000-01
Volumes
Held
Volumes
Added,
Gross
Current
Serials
Total
Expenditures
Total Prof.
& Support
Staff
1,612,448
51,126
12,452
11,552,918
118
Law
494,266
22,970
11,755
6,797,430
66
Med
162,165
2,925
1,524
1,996,448
20
2,268,879
77,021
25,731
20,346,796
204
Main
Total
Association of Research Libraries
www.arl.org
Building the Future
Main Campus Strategic Plan: 2001-2006
• Continue library enhancement so that
collection development can further support the
scholarship and teaching activities of faculty
and students.
• Integrate technology support for research and
teaching of the Library and UIS.
• Expand funding for the digital library
– Source:
http://provost.georgetown.edu/documents/implement_plan.html
Association of Research Libraries
www.arl.org
Building the future
• What do you do to build the future?
• Why is what you are doing relevant to
the future?
• How do you use the evidence you
collect from your everyday transactions
to define the future?
Association of Research Libraries
www.arl.org
What data do you collect?
What do you
collect?
Why do you
collect it?
Association of Research Libraries
www.arl.org
How do you use
it?
No Gluttony Measures
Measure what is important,
not just what is measurable
because
What you measure is what
you will pay attention to
and work toward
Association of Research Libraries
www.arl.org
Measures that Matter
• Input
Output
Outcome Impact
• Consistent with organizational mission, goals
and objectives
• Integration with program review process
• Balance customer, stakeholder, and
employee interests and needs
• Establish accountability
• Collection and use of reliable and valid data
Association of Research Libraries
www.arl.org
Opportunities and Pressures
• Increasing demand for libraries to
demonstrate outcomes/impacts in
areas of importance to institution
• Increasing pressure to maximize use of
resources through benchmarking
resulting in:
– Cost savings
– Reallocation
Association of Research Libraries
www.arl.org
Reflections …
Educational institutions today face new and
significant challenges stemming from
disruptions of financial markets,
introduction of new technologies,
demands for greater efficiency, and
unprecedented requirements for
investment in faculty, research, and
infrastructure
Some Early Reflections on TIAA-CREF by Herbert M.
Allison (February 2003)
Association of Research Libraries
www.arl.org
Why New Measures for ARL
• Increased customer and stakeholder
expectations for services and
responsiveness
• Greater demands for accountability
• Exploding growth in use and applications of
technology
• Increasing competition for resources
• Need for use of reliable and valid data
– Benchmarking and best practice
– Over time
Association of Research Libraries
www.arl.org
New Measure
How does a library answer the question,
Do We Make a Difference?
Association of Research Libraries
www.arl.org
Evaluation team representing the
Middle States Commission on Higher
Education
A number of areas emerged from an overall
observation of the many recommendations in the
student section: a concern for diversity issues,
desire for increased faculty/student interaction,
need for additional space for student use to
promote greater campus community, increased
funding and better services for graduate students,
and improved use of data from outcomes
assessment for planning.
Report to the Faculty, Administration, Trustees, Students of GEORGETOWN UNIVERSITY
Washington, DC 20057 byAn evaluation team representing the Middle States Commission on Higher
Education <http://provost.georgetown.edu/documents/evaluation.html>
Association of Research Libraries
www.arl.org
The Challenge
“The difficulty lies in trying to find a single model or
set of simple indicators that can be used by
different institutions, and that will compare
something across large groups that is by
definition only locally applicable—i.e., how well a
library meets the needs of its institution.
Librarians have either made do with
oversimplified national data or have undertaken
customized local evaluations of effectiveness, but
there has not been devised an effective way to
link the two”
Sarah Pritchard
Association of Research Libraries
www.arl.org
ARL New Measures Initiative
• Collaboration among member leaders with
strong interest in this area
• Retreat setting to define topics
• Specific projects developed with different
models for exploration
• Individual library may take the lead
• Intent to make resulting tools and
methodologies available to full membership
and wider community
Association of Research Libraries
www.arl.org
Development of Issue Papers (1)
Tuscon, January 1999
•
•
•
•
•
•
•
•
•
a) include an outcomes based definition
b) identify the need for the topic
c) provide some initial thoughts on data
d) identify what data is already collected
e) consider whether should be a part of ARL
f) recommendation for additional research
g) note whether topic is long- or short-term;
h) recommend possible experts
i) prepare a call for pilot projects
Association of Research Libraries
www.arl.org
Development of Issue Papers (2)
Tuscon, January 1999
•
•
•
•
•
Ease and Breadth of Access
User Satisfaction
Library Impact on Teaching and Learning
Library Impact on Research
Cost Effectiveness of Library Operations and
Services
• Space and Facilities
• Market Penetration
• Organizational Capacity
Source: <http://www.arl.org/stats/newmeas/nmbackground.html>
Association of Research Libraries
www.arl.org
ARL New Measures Projects
• Investigation of role libraries can play in higher education
outcomes: (a) learning (b) research
• Assessing Information Literacy Outcomes (SAILS)
• ILL/DD study to confirm changes in operations that lead
to best practices
• Project to define usage measures for electronic
information resources (E-Metrics)
• Demonstration project for service effectiveness
measures (LibQUAL+)
• Service Quality perceptions and expectations for the
digital library (e-QUAL)
Association of Research Libraries
www.arl.org
Learning Outcomes Brief History
• Engage project consultant with experience in
campus assessment activities
• Supported with funds from interested members
• Small group of leaders engaged in discussion
• Preparation of strategy paper for discussion by
larger ARL community in May 2000
• Kenneth R. Smith ‘New Roles and
Responsibilities for the University Library:
Advancing Student Learning Through
Outcomes Assessment’
• Summer 2001: Working Group
Association of Research Libraries
www.arl.org
Learning Outcomes Working
Group (1)
Summer 2001: Establishment of Learning
Outcomes Working Group
– Survey accreditation agencies to identify best
practice institutions
– Compile a list of learning outcomes being used
by ARL institutions academic departments and
general education requirements (if they exist)
– Contact professional associations to identify any
that measure learning
– Identify offerings that libraries can develop to
meet the outcomes identified
Association of Research Libraries
www.arl.org
Learning Outcomes
Working Group(2)
– Establish a plan for how to take the development
of offerings to the next stage with faculty in a
select group of institutions
– Explore collaboration with ACRL in offering
training on information literacy skills within the
ARL community
– Define or create generalized tools for
assessment
– Define learning outcomes for the purpose of this
ARL project and determine at what level the
project should focus its attention (the student,
course, program, department, etc.)
Association of Research Libraries
www.arl.org
SAILS Brief History
• Experience with SERVQUAL in many libraries over the last 10
years
• Texas A&M SERVQUAL assessment
• Meeting of interested ARL libraries (ALA Midwinter 2000)
• Pilot with 12 ARL libraries (spring 2000)
• External funding through FIPSE, U.S. Department of
Education (September 2000)
• Participation and endorsement by Big 12 Plus Consortium
• 43 libraries participating spring 2001
• 164 institutions participating in spring 2002 including OhioLINK
and AAHSL groups
• 316 institutions participating in spring 2003 including NY3Rs,
21 British libraries, 1 Dutch, and three institutions
experimenting with a French version of the instrument
Association of Research Libraries
www.arl.org
SAILS Brief History
•
•
•
•
•
•
•
•
•
Based on ACRL Standards
92 items developed
70% of ACRL learning outcomes covered
Field testing at KSU two semesters; Oregon State
Ohio Board of Regents collaborative grant with Bowling Green
State University
IMLS Grant
Association of Research Libraries endorsement
Advisory Council
10 libraries currently involved in a pilot
Association of Research Libraries
www.arl.org
ILL/DD Brief History
• ARL/RLG interlibrary loan cost study: a joint effort of the
Association of Research Libraries and the Research Libraries
Group by Marilyn M. Roche (Washington, DC: Association of
Research Libraries, 1993)
• 1996 ARL ILL/DD Performance Measures Study
– 97 research libraries and 22 Oberlin Group
– cost, fill rate, turnaround time, and user satisfaction
– Measuring the Performance of Interlibrary Loan Operations in
North American Research & College Libraries by Mary Jackson
(Washington, DC: Association of Research Libraries, 1998)
• 2003 Assessing ILL/DD Services
– 75 libraries
– Replicate performance measures study
– Add mediated and user-initiated (unmediated) component
Association of Research Libraries
www.arl.org
E-Metrics Brief History
• ARL Supplementary Statistics tracking expenditures for
electronic resources since 1993
• Facilitated retreat at Scottsdale in February 2000
• Contract with the Information Use and Management Policy
Institute at Florida State University
– Phase One: Environmental Scan
– Phase Two: Proposed Measures and Testing
– Phase Three: Training Modules
• Measures for Electronic Resources (E-Metrics) by
Wonsik ‘Jeff’ Shim, Charles McClure, and John Bertot
(Washington, DC: Association of Research Libraries, 2002)
• 2002-2003 extended pilot with 39 libraries
Association of Research Libraries
www.arl.org
Project COUNTER
• Both libraries and publishers need usage statistics
• Counting Online Usage of Networked Electronic
Resources (COUNTER)
• Gain industry support and increase awareness
• Release 1 of Code of Practice
– Focus on journals and databases
• Identify and define data elements to collect
• Usage report content, format and delivery
• Guidelines for data processing and auditing
Association of Research Libraries
www.arl.org
Project COUNTER
• Both libraries and publishers need usage statistics
• Counting Online Usage of Networked Electronic
Resources (COUNTER)
• Gain industry support and increase awareness
• Release 1 of Code of Practice
– Focus on journals and databases
• Identify and define data elements to collect
• Usage report content, format and delivery
• Guidelines for data processing and auditing
Association of Research Libraries
www.arl.org
LibQUAL+ Brief History
•
•
•
•
•
Experience with SERVQUAL in many libraries past 15 years
Texas A&M SERVQUAL assessment
Meeting of interested ARL libraries (ALA Midwinter 2000)
Pilot with 12 ARL libraries (spring 2000)
External funding through FIPSE, U.S. Department of
Education (September 2000)
• 43 libraries participating spring 2001 (Big12Plus)
• 164 institutions participating spring 2002 (OhioLINK, AAHSL) g
• 316 institutions participating spring 2003 (NY3Rs, British,
Dutch, French language version)
Association of Research Libraries
www.arl.org
LibQUAL+ Brief History
•
•
•
•
•
Experience with SERVQUAL in many libraries past 15 years
Texas A&M SERVQUAL assessment
Meeting of interested ARL libraries (ALA Midwinter 2000)
Pilot with 12 ARL libraries (spring 2000)
External funding through FIPSE, U.S. Department of
Education (September 2000)
• 43 libraries participating spring 2001 (Big12Plus)
• 164 institutions participating spring 2002 (OhioLINK, AAHSL) g
• 316 institutions participating spring 2003 (NY3Rs, British,
Dutch, French language version)
Association of Research Libraries
www.arl.org
e-QUAL Brief History
• National Science Digital Library Program at NSF
• Building on the LibQUAL+ experience
• NSF funding to modify LibQUAL+ for the digital library through
NSDL (August 2001)
• Participation in the NSDL evaluation workgroup
• Understanding NSDL Core Integration Project and distributed
collections environment
• Qualitative Grounding ongoing
• Development of a web-based tool for measuring user
perceptions and expectations of the web, digital resource,
digital collection, ?
Association of Research Libraries
www.arl.org
Future of New Measures Initiative
• Incorporation of some new data
elements in traditional surveys
• Development of workshops and
consulting services for performance
measures
• Establishment of data gathering and
statistical analysis tools
Association of Research Libraries
www.arl.org
“Grand Unified Theory”
Collaborative action +
Commitment to Learning =
Progress
Association of Research Libraries
www.arl.org
What data would support the
library’s mission?
The Georgetown University Library strives for excellence in
acquiring, organizing, interpreting, preserving, and providing
access to information resources and services that support the
University's curricular and research activities. The Library
achieves its mission by:
•
• Developing and preserving collections that support the curriculum and research
interests of the main campus faculty and students
• Organizing convenient and seamless access to print, electronic, and other
resources to facilitate research and expand scholarship
• Teaching patrons to use information resources and technologies effectively and
promoting lifelong learning skills
• Providing physical and networked environments that encourage study,
scholarship, and collaborative learning
• Retaining a staff committed to service excellence, continuous learning, and
leadership within the library profession
• Fostering collaborations throughout the University to create appropriate
partnerships and maximize resources
• Cultivating effective relationships with administrators, faculty, and students
• Cooperating regionally, nationally and internationally with other libraries and
consortia to acquire and share collections and resources with the scholarly
community.
Association of Research Libraries
www.arl.org
What data should you collect?
Association of Research Libraries
www.arl.org
Self-assessment and value
judgements
Individual
choice
Association of Research Libraries
Social
choice
www.arl.org
Measurement Tools
• Focus Groups
• Interviews
– Structured
– Unstructured
• Site Visits
• Observation
– Obtrusive
– Unobtrusive
• Case Studies
• Surveys
– Transaction
– User
Association of Research Libraries
• Critical Incident
Technologies
• User/Transaction Logs
• Networked-Based Data
Collection
• Group Process
Surveys
• Scenario Development
– Group
– Individual
www.arl.org
Methods of Assessing Students
• Standardized tests
– Pre
– Post
• Assignments
–
–
–
–
Papers and essays
Oral presentations
Demonstrations
Exhibitions
• Capstone
experiences
• Surrogates
– Grades/GPA
– Self-reports
– Interviews
• Portfolios
Association of Research Libraries
www.arl.org
Assessment Challenges
•
•
•
•
•
•
•
•
•
Resources (i.e., time and money)
Buy-in
Access to individuals to evaluate
Expertise to conduct evaluation
Project management experience
Appropriate benchmarks
Conceptual clarity
Measurement & design requirements
Instrument validity and reliability
Association of Research Libraries
www.arl.org
The Answer is in
Building the Future
Library effectiveness is becoming less
elusive and increasingly recognizable and
actionable … define the actions, the
measures, and the future before others
define them for you.
Association of Research Libraries
www.arl.org
Martha Kyrillidou, Senior Program Officer for Statistics and
Measurement, [email protected]
Consuella Askew, LibQUAL+™ Program Officer,
[email protected]
Association of Research Libraries
21 Dupont Circle, Ste 800
Washington, DC 20036
202-296-2296 (phone)
202-872-0884 (fax)
Association of Research Libraries
www.arl.org