Transcript Scholarly Communication: New Forms for a New Century?
Assessing Library Performance: New Measures, Methods, and Models
24 th IATUL Conference • 2-5 June 2003 • Ankara, Turkey Julia C. Blixrud, ARL Assistant Executive Director, External Relations
Association of Research Libraries www.arl.org
Measures that Matter
• Input Output Outcome Impact • Consistent with organizational mission, goals and objectives • Integration with program review • Balance customer, stakeholder, and employee interests and needs • Establish accountability • Collection and use of reliable and valid data
Association of Research Libraries www.arl.org
Forces and Challenges
• Increasing demand for libraries to demonstrate outcomes/impacts in areas of importance to institution • Increasing pressure to maximize use of resources through benchmarking resulting in: – Cost savings – Reallocation
Association of Research Libraries www.arl.org
ARL New Measures Projects
• Demonstration project for service effectiveness measures (LibQUAL+™) • Project to define usage measures for electronic information resources (includes institutional outcomes) • Identification of measures that demonstrate a library’s contribution to student learning outcomes • Investigation of role libraries play in support of the research process • Identification of cost-drivers and development of cost benefit studies (ILL/DocDel)
Association of Research Libraries www.arl.org
User Surveys
• Obtain direct responses to a series of questions from the community surveyed • Identify user issues, concerns and needs • Measure library performance from the user perspective, including satisfaction • Acquire quantifiable data that can be statistically analyzed and generalizeable for the larger population • Improve or change services • Increase library visibility and marketing • Contribute to broader institutional assessment/accreditation
Association of Research Libraries www.arl.org
LibQUAL+™ Project Goals
• Establishment of a library
service quality assessment program
at ARL • Development of
web-based
tools for assessing library service quality • Development of mechanisms and
protocols for evaluating
libraries • Identification of
best practices
library service in providing
Association of Research Libraries www.arl.org
Survey Instrument
3 scales:
Minimum
Desired
Perceived Association of Research Libraries www.arl.org
Dimensions of Library Service Quality
Affect of Service Empathy Responsiveness Assurance Reliability Information Access Scope Timeliness Convenience Library Service Quality Personal Control Ease of Navigation Convenience Modern Equipment Library as Place Utilitarian space Symbol Refuge Association of Research Libraries www.arl.org
Association of Research Libraries www.arl.org
Association of Research Libraries www.arl.org
1 8. 00% 1 6. 00% 1 4. 00% 1 2. 00% 1 0. 00% 8. 00% 6. 00% 4. 00% 2. 00% 0. 00% 1 993 Percentages of Acquisitions Dollars Devoted to Electronic Resources 1 994 1 995 1 996 1 997 1 998 1 999 2000 2001
Association of Research Libraries www.arl.org
ARL E-Metrics Project
Three phases: – Initial Phase (May-October 2000):
What do we know?
Inventory of current practices at ARL libraries as to statistics, measures, processes, and activities that pertain to networked resources and services.
– Second Phase (November 2000-June 2001):
What can we collect?
Identified and field tested an initial draft set of statistics and measures – Final phase (July 2001-December 2001):
What difference does this make?
Build linkages to: educational outcomes/impact, research, technical infrastructure
Association of Research Libraries www.arl.org
Recommended Statistics & Measures
• • • • • Patron Accessible Electronic Resources (R1-3) Use of Networked Resources & Services (U1-5) Expenditures for Networked Resources & Related Infrastructure (C1-3) Library Digitization Activities (D1-3) Performance Measures (P1-3)
Association of Research Libraries www.arl.org
Characteristics of Each Recommended Measure
• Definition • Rationale • Unit of Measure • Data source • Frequency • Process • Related Issues
Association of Research Libraries www.arl.org
Focus on Electronic Resources
• Project COUNTER • ICOLC • Measuring Impact (Franklin/Plum)
Association of Research Libraries www.arl.org
Cost-Effectiveness
• ILL/DD study • Technical Services Cost Study • Activity Based Costing • Balanced Scorecard • Baldrige Quality Award
Association of Research Libraries www.arl.org
Outcomes
• Institutional • Learning • Research
Association of Research Libraries www.arl.org
Learning Outcomes
• • Develop strategy to involve library in campus assessment activities to demonstrate the value of the library to the learning community – Move from content view (books, subject knowledge) to competency view (what students are able to do) – Understand learning outcomes of academic degree programs – Develop curriculum segments through which the library achieves outcomes
Information Literacy Competency Standards for Higher Education
approved by the Association of College and Research Libraries in January 2000
Association of Research Libraries www.arl.org
Standardized Assessment of Information Literacy Skills • Base Based on outcomes as defined by ACRL information literacy competency standards d on ACRL standards • Contains items not specific to a particular institution or library • Is easily administered • Has been proven valid and reliable • Assesses at institutional level • Provides for both external and internal benchmarking
Association of Research Libraries www.arl.org
Research Outcomes (Surrogates)
• R&D expenditures • PhDs awarded/year • Post-doctoral appointments • Research awards per year • Academy memberships • Faculty quality rankings
Association of Research Libraries www.arl.org
New Measure Challenges
• Resources (i.e., time and money) • Buy-in (commitment to a culture of assessment • Access to individuals to evaluate • Expertise to conduct evaluation • Project management experience • Appropriate benchmarks • Conceptual clarity • Measurement & design requirements • Instrument validity and reliability
Association of Research Libraries www.arl.org
Julia C. Blixrud Assistant Executive Director, External Relations Association of Research Libraries 21 Dupont Circle, Ste 800 Washington, DC 20036 [email protected]
202-296-2296 ext. 133 202-872-0884 (fax) 202-251-4678 (cell)
Association of Research Libraries www.arl.org