Transcript Role of Assessment in Research Libraries

The Role of Assessment
in Research Libraries
Initiative to Recruit a Diverse Workforce Leadership Symposium
January 21, 2006 • San Antonio, TX
Julia C. Blixrud, ARL Assistant Executive Director, External Relations
Association of Research Libraries
www.arl.org
Familiar Measures
• Inputs
– Collection size
– Expenditures
– Staffing
• Outputs
– Services
– People served
• Ratios (inputs  outputs)
– e.g., expenditures per FTE
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Higher Education Challenges
Educational institutions today face new and
significant challenges stemming from
disruptions of financial markets,
introduction of new technologies,
demands for greater efficiency, and
unprecedented requirements for
investment in faculty, research, and
infrastructure
Some Early Reflections on TIAA-CREF by Herbert M.
Allison (February 2003)
Association of Research Libraries
www.arl.org
Research Library Environment
• Increased customer and stakeholder
expectations for services, including
quality and responsiveness
• Greater demands for accountability
• Exploding growth in use and
applications of technology
• Increasing competition for resources
• Need for use of reliable and valid data
Association of Research Libraries
www.arl.org
Opportunities and Pressures
• Increasing demand for libraries to
demonstrate outcomes/impacts in
areas of importance to institution
• Increasing pressure to maximize use of
resources through benchmarking
resulting in:
– Cost savings
– Reallocation
Association of Research Libraries
www.arl.org
Measures that Matter
• Input --> Output --> Outcome --> Impact
• Consistent with organizational mission, goals, and
objectives
• Integration with program review
• Balance customer, stakeholder, and employee
interests and needs
• Establish accountability
• Collection and use of reliable and valid data
– Benchmarking and best practice
– Over time
Association of Research Libraries
www.arl.org
The Challenge
“The difficulty lies in trying to find a single model or
set of simple indicators that can be used by
different institutions, and that will compare
something across large groups that is by
definition only locally applicable—i.e., how well a
library meets the needs of its institution.
Librarians have either made do with
oversimplified national data or have undertaken
customized local evaluations of effectiveness, but
there has not been devised an effective way to
link the two”
Sarah Pritchard
Association of Research Libraries
www.arl.org
ARL New Measures Begins
Tuscon, AZ, January 1999
•
•
•
•
•
Ease and Breadth of Access
User Satisfaction
Library Impact on Teaching and Learning
Library Impact on Research
Cost Effectiveness of Library Operations and
Services
• Space and Facilities
• Market Penetration
• Organizational Capacity
Source: <http://www.arl.org/stats/newmeas/nmbackground.html>
Association of Research Libraries
www.arl.org
E-Metrics Brief History
• ARL Supplementary Statistics tracking expenditures for
electronic resources since 1993
• Facilitated retreat at Scottsdale in February 2000
• Contract with the Information Use and Management Policy
Institute at Florida State University
– Phase One: Environmental Scan
– Phase Two: Proposed Measures and Testing
– Phase Three: Training Modules
• Measures for Electronic Resources (E-Metrics) by
Wonsik ‘Jeff’ Shim, Charles McClure, and John Bertot
(Washington, DC: Association of Research Libraries, 2002)
• 2002-2003 extended pilot with 39 libraries
• Revised supplementary statistics data collection
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Learning Outcomes
• Development of strategy for involving library in
campus assessment activities to demonstrate the
value of the library to the learning community
– Move from content view (books, subject knowledge) to
competency view (what students are able to do)
– Understand learning outcomes of academic degree programs
– Develop curriculum segments or “offerings” through which the
library achieves outcomes
• Information Literacy Competency Standards for
Higher Education approved by the Association of
College and Research Libraries in January 2000
Association of Research Libraries
www.arl.org
Association of Research Libraries
www.arl.org
Project SAILS
• Developed by Kent State University
• Based on ACRL Standards
• IMLS Grant as well as Ohio Board of Regents
collaborative grant with Bowling Green State
University
• 3 year research project involving 80 institutions and
more than 42,000 students
• Measures cohorts of students
• Benchmarking and comparative reports on skill sets
Association of Research Libraries
www.arl.org
Assessment’s Purpose
How can a library answer the question,
Do We Make a Difference?
Association of Research Libraries
www.arl.org
Rise of User-Centered
Library and the Culture of
Assessment in the 1990s
User-Centered Library
Culture of Assessment
• All services and activities
are viewed through the eyes
of the customers
• Customers determine
quality
• Library services and
resources add value to the
customer
• Organizational environment
in which decisions are
based on facts, research
and analysis,
• Services are planned and
delivered to maximize
positive customer outcomes
Association of Research Libraries
www.arl.org
Culture of Assessment
Key Elements
• Basic value - customer & learning focus
• A Culture of Assessment is an organizational
environment in which decisions are based on facts,
research and analysis, and where services are planned
and delivered in ways that maximize positive outcomes
and impacts for library clients
• A Culture of Assessment exists in organizations where
staff care to know what results they produce and how
those results relate to customer expectations
• Organizational mission, values, structures, and systems
support behavior that is performance and learning
focused
Association of Research Libraries
www.arl.org
Why Do Libraries Need a
Culture of Assessment?
• Role within the parent organization
– Relationship to central mission
– Accountability for operations, resources, added-value
• Need for efficiency and effectiveness of operations
– Management of resources
– Decision-making based on data
– Institutionalization of planning process
• Response to customers
– High quality service
– Focus on added value
Association of Research Libraries
www.arl.org
Important Characteristics
• Leadership has sense of purpose, urgency, resolve, and
flexibility
• Organizational focus is on customers
• Feedback is welcomed and used (atmosphere of
integrity and trust
• Staff care about outcomes and impact
• Environment is one in which facts are analyzed and
research is conducted
• Staff are learning how to measure accurately from the
customers’ point of view
• Organization can anticipate future needs
• Organization is building relationships with customers
Association of Research Libraries
www.arl.org
In Building a Culture of Assessment
-- We Often Have a GAP
Association of Research Libraries
www.arl.org
The Importance of
Appropriate Measures
Measure what is important,
not just what is measurable
because
What you measure is what
you will pay attention to
and work toward
Association of Research Libraries
www.arl.org
Performance Management Maxim
If you can’t measure it, you
can’t manage it.
What gets measured matters.
Association of Research Libraries
www.arl.org
Issues in Using Data Effectively
•
•
•
•
•
•
•
•
•
Library leadership
Organizational culture
Priorities of the library
Sufficiency of resources
Data infrastructure
Assessment skills and expertise
Sustainability
Presenting results
Using results to improve libraries
Association of Research Libraries
www.arl.org
Choosing the Right Method
•
•
•
•
•
•
•
Appropriate for the information needed
Timely
Cost effective
Level of user involvement
Representativeness of population
Support for staff/training available
Possiblity/probability for results to lead to
positive change
Association of Research Libraries
www.arl.org
Quantitative Measurement Tools
• Surveys
– Employee survey
– Total market survey
– Transaction-based
questionnaires
– User survey
• Internal recordkeeping
• Service data capture
Association of Research Libraries
• Transaction logs
• Survey methods
–
–
–
–
www.arl.org
Email
Paper
Telephone
Web-based
Qualitative Measurement Tools
•
•
•
•
Advisory teams
Complaint system
Customer visit teams
Employee field
reporting
• Employee visit teams
• Focus groups
• Mystery shopping
service
Association of Research Libraries
•
•
•
•
•
•
•
•
Observation
Portfolios
Service reviews
Spot comment cards
Structured interviews
Toll-free hotlines
Usability studies
User groups
www.arl.org
Methods of Assessing Students
• Standardized tests
– Pre
– Post
• Assignments
–
–
–
–
Papers and essays
Oral presentations
Demonstrations
Exhibitions
• Capstone
experiences
• Surrogates
– Grades/GPA
– Self-reports
– Interviews
• Portfolios
Association of Research Libraries
www.arl.org
Multiple Methods Provide More
Effective Measurement
•
•
•
•
•
•
•
Complementary
Appropriateness
Large projects can be divided up
Quantitative and qualitative information
Multi-dimensional views of issues or users
“Two Proofs” (cross validation)
Use of existing data
Association of Research Libraries
www.arl.org
Barriers to Using Data
Effectively in Libraries
•
•
•
•
•
•
•
Organizational culture/leadership support
Time/Staff/Resources
Data issues – too much, compatibility, validity
Establishing priorities
Knowing what to measure and methods to use
Inexperience, perceived lack of skills and expertise
Understanding, presenting and knowing what to do
with the results
Hiller, S. and Self, J. (2004). From Measurement to Management: Using
Data Wisely for Planning and Decision-Making. Library Trends.
Association of Research Libraries
www.arl.org
Statistics are no substitute
for judgment
-- Henry Clay
Association of Research Libraries
www.arl.org
Assessment Challenges
•
•
•
•
•
•
•
•
•
Resources (i.e., time and money)
Buy-in
Access to individuals to evaluate
Expertise to conduct evaluation
Project management experience
Appropriate benchmarks
Conceptual clarity
Measurement & design requirements
Instrument validity and reliability
Association of Research Libraries
www.arl.org
Julia C. Blixrud
Director of Information Services
Association of Research Libraries
21 Dupont Circle, Ste 800
Washington, DC 20036
[email protected]
202-296-2296 ext. 133
202-872-0884 (fax)
202-251-4678 (cell)
Association of Research Libraries
www.arl.org