Transcript Assess for Success: Proving Library Value Steve Hiller Director, Planning and Assessment University of Washington Libraries.

Assess for Success: Proving Library Value
Steve Hiller
Director, Planning and Assessment
University of Washington Libraries
Assessment Basics
• What do we need to know about our communities and
customers to make them successful?
• How do we measure the effectiveness of our
services, programs and resources?
• What do our stakeholders need to know in order to
provide the resources needed for a successful library?
Assessment Is More than Numbers
Library assessment is a structured process:
• To learn about our communities
• To respond to the needs of our users
• To improve our programs and services
• To contribute to the success of our communities
Assessment Process
• Focuses on understanding customer needs and offering
services that meet those needs
• Collects, analyzes and uses quantitative and
qualitative data for more effective management and
decision-making
• Emphasizes ongoing communication with customers
and stakeholders,
• Seeks opportunities for collaboration and comparisons
within the organization, institution and beyond
What Do We Need to Know About Our
Customers?
• Who are our customers (and potential customers)?
• What areas/fields/courses are they working in?
• How do they work? What’s important to them?
• How do they find information needed for their work?
• How do they use our services? What would they change?
• How do they differ from each other in use/needs?
• Outcomes
How do we add value to their work?
How do we contribute to their success?
The Value of Community Assessment
•
•
•
•
•
•
•
Identify actual and potential customers
Understand needs and use preferences
Use funding and staff effectively
Understand and address “competition”
Encourage community involvement and “ownership”
Target marketing, market penetration and outreach
Measure, demonstrate, present the value of the library
to the community and stakeholders
Thinking Strategically About Library Futures
• What is our central work and how can we do more,
differently, and at less cost?
• What important services do we provide that others
can’t?
• What advantages do we possess?
• How is customer behavior changing?
• How do we add value to our customers work?
• What are the essential factors responsible for library
success now and in the future?
The Challenge for Libraries
• Traditional statistics and data are no longer sufficient
–
–
–
–
Emphasize inputs – how big and how many
Do not tell the library’s story
May not align with organizational goals and plans
Do not measure service quality
• Need data from the user’s perspective
• Need to provide organizational accountability to
funders and other stakeholders
• Need to define where we want to be in 3/5/more years
• Need to answer: What value do we provide our
primary community and how do we show it?
Defining Success in a Digital Environment
• Crafting new indicators and measures of success
• Moving from measuring inputs to outputs and
outcomes
• Understanding impact of library/partner roles and
services on the community
• Agreeing on qualitative measures of success: user
perceptions, user success, creating value, advancing
community goals.
• Reallocating resources and managing capabilities to
achieve success.
• Demonstrating value of library contributions
Some New Metrics for Academic Libraries
•
•
•
•
•
•
•
•
Uniqueness of collections
Value of consortia
Administrative and budgetary efficiencies
Student outcomes/student learning/graduate success
Contributions to faculty productivity
Social frameworks/intellectual networks
Generating new knowledge
Creating the collective good with reusable assets such
as an institutional repository or e-science
Good Assessment Starts Before You Begin . . .
Some Questions to Ask
• Define the question
– What do you need to know and why
• How will you use the information/results
• Where/how will you get the information
–
–
–
–
What are appropriate methods
Is there existing information
New data (where or who will you get it from)
Is it cost and resource effective
• How will you analyze the information
• Who will act upon the findings
How Do We Get Customer Information?
•
•
•
•
•
•
•
•
•
Statistics/data mining (local, institutional)
Surveys
Focus groups
Observation
Usability
Interviews
Embedding
Logged activities
Comments, suggestions, “over the counter”, being there
Another View of Metrics
Getting The Right Data Isn’t Enough
“…but to suppose that the facts, once
established in all their fullness, will ‘speak
for themselves’ is an illusion.”
Carl Becker
Annual Address of the President of the American Historical Association, 1931
Understanding what the data means and how to
follow-up are crucial
Make Data Meaningful
•
•
•
•
•
Summarize
Compare
Analyze
Present
Go below the surface to examine results by:
– Demographic group
– Users and non-users
– Stakeholders vs non-stakeholders
• Compare current data with past; look for trends
• How can we use the data for action?
Data Presentation: 3 Key Questions
• What’s the message?
– Fewer “messages” means greater impact
• Who’s the audience?
– Multiple audiences may need different presentations
• How do we present?
– Quantitative data - Be graphic , provide meaning/understanding
– Qualitative data - Be selective, provide corroborating data
• Keep it simple AND focused!
And Remember the Content!
Moving from Analysis to Action
• From all of the data, determine what can and should be
addressed
• Prioritize the important/critical action items
– Make evidence-based decisions
– Align with mission, vision and goals of parent organization
– Address what is important to customers
• Establish action guidelines
– Who, how, when
What Makes it Hard to Be Evidence-Based?
(From Pfeffer and Sutton, 2006)
•
•
•
•
•
•
•
There’s too much evidence
There’s not enough good evidence
The evidence doesn’t quite apply
People are trying to mislead you
You are trying to mislead you
The side effects outweigh the cure
Stories are more persuasive anyways
When the Evidence Isn’t Used
Collection Assessment: The 4 Questions
What’s important from the:
Library perspective?
Customer perspective?
Stakeholder perspective?
Vendor/publisher perspective(s)?
Some Good Reasons to Assess Collections
•
•
•
•
•
•
•
•
Meeting user needs
Costs/Budget/Space
Changes in the information discovery/use processes
Accountability to stakeholders, funding agencies
Breadth and depth; comprehensiveness
Environmental changes (social, information, technical)
Changes in publisher/vendor marketing and packaging
Return on Investment (ROI); contingent valuation
What Numerical/Cost Data Do We Need?
• Costs
– Collections (annual/over time)
• Titles
• Packages, consortia
– Operational
• Staff
• Facilities
• Vendors
• Usage
– Print (on-site, check-out, ILL)
– Electronic/online
• Type of use
– Packages, consortial, doc delivery
– Change over time
• Cost per use
– What type of use
•
•
•
•
Downloads
Searches
Loans
Page views
– Packages, individual titles
• Global information
– Collection comparisons
– Publishing output
– Bibliometric tools, citation
analyses, collection
– Community demographics
So We Need Systems/Products That:
Enable us to track and validate usage
Enable us to track item level and package level costs
Provide comparative information
Provide “canned”/custom reports easily
• Provide data in flexible formats we can manipulate/analyse
• Provide information that meets established standards
• Present snapshot and trend data
• Are cost-effective
E-Metrics Tools: COUNTER, SUSHI and ERMI
• Counting Online Usage of NeTworked Electronic
Resources
– Standardized way to count usage
– Defined reports for journals, databases, and eBooks
• Standardized Usage Statistics Harvesting Initiative
– Model for automation of statistics harvesting
– SUSHI feed: usage data that magically appears!
• Electronic Resources Management Initiative
– Can accept SUSHI feeds
– Easy to generate COUNTER-style reports within ERM module
– Brings in data from order records to calculate cost per use
Genesis Statistics Detail
2007
2007
2008
2008
Title
CPU
Use
CPU
Use
Genesis
$3.00
649
$3.67
93
2007 payment / 2007 Use = 2007 CPU
$1950.71 / 649
= $3.00
Don’t Forget the Qualitative
•
•
•
•
Discovery pathways and user expertise
Usability
User needs
User outcomes
– What did the information enable the user to accomplish?
Integrated Collection Assessment
• Collections-centered
– Size, growth, formats, authors, subjects, comparisons
• Cost-centered
– Expenditures, price, inflation, cost per use, return on
investment
• Use-centered
– Loans, e-usage, purchase requests, ILL
• Customer centered (secondary)
– Community demographics, programs, research, bibliometric
studies, consortial
• Customer centered (primary)
– Behavior, impact, outcomes, success, discovery
What Will We Do With the Information?
•
•
•
•
Analyze and understand
Present and share
Make good decisions and choices
Ensure we add value to the work of our community
Customers and stakeholders make the ultimate
determination of value!
Focus on the UW Libraries Contribution
to the Research Enterprise
• How does the library support sponsored research?
• What is the value of the library contribution?
• Some measures & indicators of library contributions
–
–
–
–
–
–
–
–
–
Collections budget and usage data
Survey information
Bibliometrics
Frequency of use
Time saved/contingent valuation
Information seeking behavior
Qualitative information
Services used
Library research projects/collaborations
The Importance of the Research Enterprise
University of Washington Operating Revenues
$2.4 Billion in 2007-08
Other
11%
Gifts
8%
Research Grants
45%
Investment
Income 3%
State funding
15%
Tuition
18%
Research Grants $1.1 Billion
Health and Human Services
$510 million
National Science Foundation
$80 million
Other federal agencies
$200 million
Industry/Foundations
$110 million
Other non-federal
$160 million
University of Washington Libraries
Assessment Methods Used
• Large scale user surveys every 3 years (“triennial
survey”) beginning in 1992. In 2007:
– 1500 faculty responses; 600 grad; 500 undergrad
• In-library use surveys every 3 years beginning 1993
– 4000 surveys returned in 2008
• Focus groups/Interviews (annually since 1998)
• Observation (guided and non-obtrusive)
• Usability
• Usage statistics/data mining
Information about assessment program available at:
http://www.lib.washington.edu/assessment/
The Changing Business Model: Trends in
Library Use at the University of Washington
Traditional Library Core
Business (Usually in-person)
UW Libraries Usage Data
• Physical Collections
800,000 in 2002-03
300,000 in 2007-08
(6.0 million article downloads in 2007-08)
– Print (primarily)
– Microforms
– Other
• Facilities
– House collections
– Customer service & work space
– Staff work space
• Services
– Reference
– Instruction
– Access
Items Used In-Library
Gate Counts
4.6 million in 2002-03
4.3 million in 2007-08
(8.7 million Web site user sessions in 2007-08)
In-Person Reference Questions
140,000 in 2002-03
90,000 in 2007-08
(18% of all ref queries are virtual in 2007-08)
34
Qualitative Information Provides Context
BioScience Interview/Focus Groups (2006)
• Content is primary link to the library
– Identify library with ejournals; want more titles & backfiles
• Provide library-related services and resources in our
space not yours
– Discovery begins primarily outside of library space with
Google and Pub Med; Web of Science also important
– Library services/tools seen as overly complex and fragmented
• Print is dead, really dead
– If not online want digital delivery
– Go to physical library only as last resort
• Personal connection is important
UW Faculty Mode of Use by Academic Area
1998/2007 (w eekly or more often)
Non- Weekly
17%
Rem ote
Only
45%
Non- Weekly
5%
Rem ote
Only
87%
Rem ote &
Visit
7%
0%
Health Sci
1998
Non- Weekly
25%
Rem ote
Only
26%
Health Sci
2007
Visit Only10%
Non- Weekly
15%
Rem ote
Only
23%
Non- Weekly,
9%
Rem ote
Only
47%
Rem ote
Only
72%
Rem ote &
Visit
51%
Rem ote &
Visit
39%
Rem ote &
Visit
32%
Visit Only 6%
Non- Weekly
9%
Rem ote &
Visit
42%
Rem ote &
Visit
17%
1%
Science-Engin Science-Eng
1998
2007
Visit Only
10%
Hum-Soc Sci
1998
1%
Hum-Soc Sci
2007
36
Off-Campus Remote Use 1998-2007
(Percentage using library services/collections at least 2x week)
70%
60%
70%
76% of faculty (80% of those using federal research
funds) connect online at least 2x week
Grad
60%
50%
50%
Faculty
40%
40%
30%
30%
20%
Undergrad
20%
10%
10%
0%
0%
1998
2001
2004
2007
Primary Reasons for Faculty Use of Libraries Web
Sites by Academic Area (2007 Triennial Survey, at least 2x per week)
75%
65%
55%
45%
35%
25%
15%
Health Sci
Library Catalog
Science-Engin
Bib Database
Hum-Soc Sci
Online journal articles
Theabiltyocsfu-xrpdeahtilonugbray
subcriptonmyaefhlbrndisctomyeah.
NeurobilgyGadStn
Importance of Books, Journals, Databases
byAcademic Area Scale of 1 “not important” to 5 “very important)
4.8
Journals
>1985
4.6
4.4
4.2
Books
4
Journals
<1985
3.8
3.6
Bib
Databases
3.4
3.2
Health Sciences
Books
Science-Engineering
Journals<1985
Bib Databases
Hum-Soc Science
Journals>1985
UW Libraries Serial Purchasing and Usage
• 85% of collections budget spent on serials/backfiles
– Ranked 4th in 2006-07 ARL serial expenditures ($11.4 million)
• 6 million journal article downloads (Counter compliant)
– 30% increase over 3 years
– 75% of downloads from scholarly journals in scienceengineering-health sciences
• Bibliographic database use has declined
– Most pronounced in subject specific databases
– Web of Science unchanged at 160,000 login sessions annually
(13% of all log-in sessions)
UW Libraries Contribution to
Sponsored Research
Among faculty receiving federal research funding:
• 97% rate Libraries as very important to their work
• 96% rate journals as most important info resource for
their work
• 93% rate the Libraries as making a major contribution
to keeping current in their field
• 93% rate the Libraries as making a major contribution
to their research productivity
• 80% connect to the Libraries at least twice per week
from office, lab or off-campus
Differences in Library Contribution by Funding
Source:
(Scale of 1 “Low to 5 “High”)
Productive research
Keeping current in
your field
Fed Funding
No fed funding
Overall Importance
to work
Journal importance
4
4.2
4.4
4.6
4.8
5
Libraries Contribution to Your Being a More
Productive Researcher by Academic Area and
Funding Source
5
No federal research funds
Federal research funds
4.75
4.5
4.25
4
All
Humanities-Soc Sci Science-Engineering
Health Sciences
Closing the Loop:
Success with Assessment
•
•
•
•
•
•
•
Assess what is important
Keep expectations reasonable and achievable
Use multiple assessment methods; corroborate
Mine/repurpose existing data
Focus on users; how they work, find & use information
Use the data to improve and add customer value
Keep staff, customers and stakeholders involved and
informed
Eye to the Future
Measuring performance is an exercise in measuring the
past. It is the use of that data to plan an improved future
that is all important. Peter Brophy
• Data trends can inform the future
• Strategic planning can frame the future
• Organizational performance models can align ongoing
operations with future aspirations
• Understanding how customers work and how that work is
changing is key to our future