Transcript Slide 1

Why assess? What is assessment?
What do we mean by actionable data?
Martha Kyrillidou
Planning for Results: Making Data Work for You
The Cato Institute, Washington, DC
February 20, 2008
What’s bringing us together …
Association of Research Libraries
http://www.arl.org/stats
Overview
1. Initiate
(Why do we assess?)
2. Plan
(What do we assess?)
3. Design
“Why assess? What is assessment? What do we
mean by actionable data?” - Martha Kyrillidou (1&2)
“Survey Design” – Neal Kaske (2&3)
“Performance Measures” – Rick Garcia (2&3)
(How do we assess?)
4. Implement
“Analysis and Interpretation” - Ray Lyons (5)
(Design in action)
5. Analyze and Interpret
(What do the results mean?)
LibQUAL+®, MINES™, and Other Tools; Packaging
Results to Communicate to Stakeholders
Gary Roebuck and Kristina Justh (2,3,4,&5)
6. Reflect and Initiate
(What do we do with the results?)
Association of Research Libraries
http://www.arl.org/stats
Clear and present danger
•
Bangor University considers removing librarians posted by Blake on
Thursday January 27, 2005 @07:30AM -753 hits
Ms Information writes "News from the University of Wales Bangor in the
UK. senior management no longer feel that subject librarians / academic
liaison librarians are needed in the modern academic library. They have
made restructuring proposals which include removing all but one of the
subject librarians and a tier of the library management, including the Head
of Bibliographic Services. The university management thinks that
technology has 'deskilled' literature searching. As far as I know, this
proposal is unprecedented in the United Kingdom. In essence, there will
remain 4 professional librarians serving a 'research-led' university of 8,000
plus FTEs and with 8 library sites. These will be the university librarian,
cataloguing librarian, acquisitions librarian and Law librarian.
•
Has anything like this happened anywhere that you know of? If so, what
have been the effects?
Association of Research Libraries
http://www.arl.org/stats
Assessment
Librarian
Library of the Future Will Also Need . . .
. . . To have it’s own data collection and management
personnel, individuals who constantly collect, analyze
and prepare reports on data regarding what services
are being used, which portions of the collection are
getting the highest usage, what materials are being
lent through interlibrary loan, and who patrons are.
Documenting the libraries contributions to quality
teaching, student outcomes, research productivity will
become critical.
Association of Research Libraries
http://www.arl.org/stats
Technical Skills needed
• Quantitative Analysis: SPSS
• Qualitative Analysis: Atlas.ti
• Data collection: Custom Web interface, Survey
Monkey, FormSpring
• Data storage: SQL Server, Excel
• Reporting: Crystal Reports, SPSS, SQL
Reporting Services, Excel, Word
• Web delivery: Dundas Charts
• Web development: ASP.NET, C#, ColdFusion
Association of Research Libraries
http://www.arl.org/stats
Communication Skills needed
• Communicating organizational vision and
service ethic
• Leadership in capturing the future and making
it happen
• Ability to take initiative and willingness to take
risks and learn from mistakes
• Troubleshooting and being an independent
thinker
• Ability to work collaboratively and move
forward team and organizational goals
Association of Research Libraries
http://www.arl.org/stats
. . . chart a course for every endeavor that
we take the people's money for, see how
well we are progressing, tell the public how
we are doing, stop the things that don't
work, and never stop improving the things
that we think are worth investing in.
– President William J. Clinton, on signing the Government Performance
and Results Act of 1993
Association of Research Libraries
http://www.arl.org/stats
What’s driving the ‘assessment’ agenda
• “Times are changing”
– Exploding growth in use and applications of
technology
– Increased customer expectations for services,
including quality and responsiveness
• Shrinking budgets
– Justification for spending $$$ on resources,
programs, and services
– Increasing competition for resources
– Cost savings and/or reallocation
– Fighting to remain relevant
Association of Research Libraries
http://www.arl.org/stats
The Internet Goes to College
Early data from ethnographic interviews
– “I use Google because I heard it searches for more
things” (than other sources).
– “I believe I can find anything on the Internet. There
hasn’t been anything I haven’t been able to find.”
– “Because I’m lazy.”
– Books have “so much information that no one can go
through it all.”
– I use “the Internet first because it is more convenient.”
– I go to the library “because that’s what teachers like.”
– “Google has gotten me through college.”
Source: Steve Jones, The Internet Goes to College, ARL Talk
Association of Research Libraries
http://www.arl.org/stats
Libraries Remain a Credible Resource in
21st Century
98% agree with statement, “My … library
contains information from credible and
known sources.”
Note. Digital Library Federation and Council on Library and Information
Resources. (2002). Dimensions and Use of the Scholarly Information Environment.
Association of Research Libraries
http://www.arl.org/stats
OCLC Reports
• Sharing, Privacy and Trust in Our Networked
World
• College Student’s Perceptions: Libraries and
Information Resources
• Perceptions of Libraries and Information
Resources
» OCLC: http://www.oclc.org/reports/sharing/default.htm
Association of Research Libraries
http://www.arl.org/stats
‘Universal’ library brand
• Amazing stability of library user’s desired
expectations across
–
–
–
–
Types of users
Languages
Countries
Some differences across disciplines
• 297,158 LibQUAL+® survey respondents
Association of Research Libraries
http://www.arl.org/stats
Towards what end
The emerging library user sees the need for a different
kind of library – a library that serves as a source of
information ready to be accessed any time, from
anywhere, using modern technologies. Can we
effectively translate the comfort and assurance of
human interactions into a virtual reality that is
trustworthy, reliable, and ever present? And will
learning be quite as effective or deep in this virtual
world of never-ending interaction with the collective
digital-knowledge base?
• Thompson, Kyrillidou, Cook “Library Users’ Service Desires”
Library Quarterly (2008)
Association of Research Libraries
http://www.arl.org/stats
Library Assessment and
its Global Dimensions
• Markets and people exposed to economic and
social frameworks unheard of before
• Competing internationally
• Library users exposed to global forces
• Libraries facing similar challenges
• Libraries as the Internet
• Libraries as Google
• Libraries as Collaborative Spaces
Association of Research Libraries
http://www.arl.org/stats
Defining Success in a Digital Environment
• Crafting new measures of success.
• Moving from measuring inputs to outputs and
outcomes
• Understanding impact of library roles and services.
• Agreeing on qualitative measures of success: user
perceptions, user success, creating value,
advancing organizational goals.
• Reallocating and managing capabilities to focus
on new definitions of success.
Association of Research Libraries
http://www.arl.org/stats
Thinking Strategically About Libraries’
Futures
• What is the central work of the library and how can we do
more, differently, and at less cost?
• What important set of services does the library provide that
others can’t? What new roles are needed?
• What advantages does the library possess?
• What will be the most needed by our community of users in
the next decade? How is user behavior changing?
• What should our libraries aspire to be ten years from now?
What are the implications of technology driven change?
• What are the essential factors responsible for the success
of the library?
Association of Research Libraries
http://www.arl.org/stats
Library Assessment
Library assessment provides a structured process to
learn about our communities, their work and the
libraries connection to what they do
The information acquired through library assessment is
used in an iterative manner to improve library
programs and services and make our libraries
responsive to the needs of our communities.
Academic libraries do not exist in a vacuum but are
part of a larger institution. Assessment within the
institution may take place in individual areas as well
as at the broad institutional level.
Association of Research Libraries
http://www.arl.org/stats
Assessment is…
“…a structured process to learn about
communities and evaluate how well the
library supports them.”
- Steve Hiller, Director of Assessment & Planning
University of Washington Libraries
Association of Research Libraries
http://www.arl.org/stats
Free speech wall,
Charlottesville, Sept 2006
Association of Research Libraries
http://www.arl.org/stats
Evaluation – foci by Peter Brophy
We have 2
million
records
Inputs
We have the
technology to
transform data
into informationrich products
Processes
26
students
have read
this paper
Outputs
Outcomes
As a result
of our
service,
humankin
d has
beaten
malaria
Impacts
We have
delivered
one million
documents
Increasing difficulty …………
Increasing usefulness …………
Association of Research Libraries
http://www.arl.org/stats
IFLA: Measuring Quality
• Resources, infrastructure: What services does
the library offer?
• Use: How are the services accepted?
• Efficiency: Are the services offered costeffectively?
• Potentials and Development: Are there
sufficient potentials for future development?
(an adaptation of the Balanced Score Card)
Association of Research Libraries
http://www.arl.org/stats
Assessment at ARL
• A gateway to assessment tools: StatsQUAL®:
– ARL Statistics -- E-Metrics
– LibQUAL+® -- Service Quality and Satisfaction
– DigiQUAL®
– MINES for Libraries®
•
•
•
•
•
Library Assessment Conferences – libraryassessment.org
Library Assessment Blog – libraryassessment.info
SPEC Kits
Service Quality Evaluation Academy
ESP Assessment (Making Library Assessment Work)
– Effective, Sustainable, Practical
Association of Research Libraries
http://www.arl.org/stats
Qualitative Profile
Developing New Metrics (per Yvonna Lincoln)
•
•
•
•
•
•
•
•
Uniqueness of collections
Defining the value of consortia
Administrative and budgetary efficiencies
Student outcomes/student learning/graduate success
Contributions to faculty/stakeholders productivity
Social frameworks/intellectual networks
Generating new knowledge
Creating the collective good with reusable assets
Association of Research Libraries
http://www.arl.org/stats
How do you describe a Library?
•
•
•
•
•
•
•
•
Breadth and quality of collections and services
Sustained institutional commitment to the library
Distinctive resources in a variety of media
Services to the scholarly community
Preservation of research resources
Contributions of staff to the profession
Effective and innovative use of technology
Engagement of the library in academic planning
Association of Research Libraries ‘Principles of Membership’
Association of Research Libraries
http://www.arl.org/stats
Group exercise
• Share important aspects of your
–Services
–Collections
–Collaborative Relations
Association of Research Libraries
http://www.arl.org/stats
Tell the story …
•
•
•
•
•
•
•
•
Facilities: number and type (library buildings, classrooms, studios, computer
labs, cafes, offices outside of the library, etc.)
University centers, services and responsibilities (managed by the library or
housed in the library): slide libraries, media centers, teaching and learning
centers, education centers, writing centers, distance ed or online course
development/instructional design, iTunesU, laptop support, technology help
desk, escholarship centers, university press or other publishing entity,
university archives, university records management, etc.
Digital collections: institutional/digital repositories, images, articles,
simulations, data, learning objects, etc.
Grants and development: numbers, types, friends organizations, advisory
boards/councils, etc.
Partnerships: nature and number, on campus and external
Consortial memberships and roles
Outreach: web hits, computer use hours, learning hours/contact/impacts (in
classrooms, Info Lit, etc.), visits, external teaching, volunteering, programs
offerings, sponsored lectures, etc.
Leadership: on campus and external committees, boards, initiatives, projects,
etc. Publications/scholarship: numbers, types (articles, books, presentations,
poster sessions, performances, etc.)
» Email from ARL Library Director – February 13, 2008
Association of Research Libraries
http://www.arl.org/stats
Measuring Impact
• Concerns Based Adoption Model (CBAM)
– Stages of concern
• How engaged are subjects in the service /innovation?
• How much interest do they show?
– Levels of use
• How much use do they make?
• Avoid dichotomy of user / non-user
– Levels of Impact
• Draw together concern and use
• Assess effects on behaviour
» Peter Brophy (South Africa, August 2007)
Association of Research Libraries
http://www.arl.org/stats
Measuring Impact
• Surface learning
– Skim, memorise, regurgitate for tests
• Little long-term impact
• Deep learning
– Involved, questioning, interactive, integration with own
knowledge
• Significant long-term impact
• A habit of critical thinking
» Peter Brophy (South Africa, August 2007):
Association of Research Libraries
http://www.arl.org/stats
“If you think you're too small to
have an impact, try going to bed
with a mosquito.”
»Anita Roddick
Association of Research Libraries
http://www.arl.org/stats
Actionable Data
• What is the desired outcome of assessment?
– An organization that strives to be customer-centered
and acts on information about environmental factors
and especially on user input.
» Raynna Bowlby and Martha Kyrillidou
Association of Research Libraries
http://www.arl.org/stats
Align with mission, vision and goals of
parent organization
– Are there prominent themes/directions – research
support, instruction, physical facilities; priority user
groups; expansion and growth or conservation and
reallocation of resources; emphasis on current
technologies and methods or more traditional
approaches
– Are there strategically favored programs/initiatives
and/or individual champions with key agendas?
– Are there stated goals for the organization and the
library?
Association of Research Libraries
http://www.arl.org/stats
Address users’ top priorities
• Are you undertaking initiatives that users indicate are
their highest wants and are you doing enough?
• What budget/resources are being re/allocated to these
initiatives?
• Are enough budget/resources being directed to these
initiatives?
• Have the departments/staff that most affect these
initiatives been informed?
• Do these departments/staff understand the data (and
comments/specifics) and have goals and improvement
plans in place?
• Are there improvement targets; how is change being
measured?
Association of Research Libraries
http://www.arl.org/stats
Identify resources that can be reallocated
• Is the work that you are performing addressing areas of
strong dissatisfaction?
• Are you doing too much of what you’ve always done,
rather than the work that results in user wants & needs?
• What budget/resources are being allocated to legacy
work that is not a user priority and is it too much?
• Have the departments/staff that are currently doing this
work been informed of changing user wants and needs?
• Do these departments/staff understand the data (and
comments/specifics) and can they redirect their time and
resources?
• Are there improvement targets; is change being
measured?
Association of Research Libraries
http://www.arl.org/stats
Continuous improvement and
assessment
• How will you incorporate the results of your analysis
and findings into your Library’s strategic and annual
plans?
• How can you develop some of your more resourceintensive improvements over several years?
• How can you insure that your assessment results are
always available when and where decisions are being
made?
• How and when will you be assessing the user
improvements implemented after your user survey?
Association of Research Libraries
http://www.arl.org/stats
The importance of accurate data in decision making
Association of Research Libraries
http://www.arl.org/stats
Your library …
• Organization and use of information is focused on
disciplines and increasingly interdisciplinary centers
• Subject-oriented staff provide information services
required and integrate them in the research process
• Developing and maintaining relationships with key
stakeholders in the disciplines/departments
• Acquainting specialized stakeholders with the broad
range of information and information services available
outside their specialty
• Assuring adequate performance of network and
technology based information services
• Assuring adequate performance of the centralized basic
support functions
Association of Research Libraries
http://www.arl.org/stats
Exercise
• What are the key indicators of this library's
work?
• How do you monitor each indicator? How do
you record the data for each indicator? With
what frequency?
• To whom and how do you communicate
information about the library's performance?
Association of Research Libraries
http://www.arl.org/stats
Association of Research Libraries
http://www.arl.org/stats
Making Library Assessment Work
• ARL project approved in 2004 and funded by
participating libraries
• Site visits by Steve Hiller (U of Washington) and Jim
Self (U of Virginia)
– Presentation
– Interviews and meetings
– Report to the Library
• 24 libraries in U.S. and Canada visited in 2005-06
• Effective, Sustainable and Practical Library
Assessment in 2007
– Open to all libraries
– 6 libraries participated in 2007
Association of Research Libraries
http://www.arl.org/stats
What We Found
• Strong interest in using assessment to improve customer
service and demonstrate value of library
• Many libraries uncertain on how to establish, maintain,
and sustain effective assessment
• Effectiveness of assessment program not dependent on
library size or budget
• Each library has a unique culture and mission. No “one
size fits all” approach works.
• Strong customer-focus and leadership support were
keys to developing an effective and sustainable
assessment
Association of Research Libraries
http://www.arl.org/stats
What are the lessons learned?
• Understanding changes in users approach to information
resources.
• Service quality improvement is a key factor.
• Understanding the impact of e-resources on library
services - TRL.
• Learning how to compete or collaborate with Google.
• Upfront investment in design and development.
• Making the assessment service affordable, practical, &
effective.
• Assessment needs to be satisfying and fun.
Association of Research Libraries
http://www.arl.org/stats
Multiple methods
•
•
•
•
•
•
•
•
•
•
•
Transactional surveys*
Mystery shopping
New, declining, and lost-customer surveys
Focus group interviews
Customer advisory panels
Service reviews
Customer complaint, comment, and inquiry capture
Total market surveys*
Employee field reporting
Employee surveys
Service operating data capture
*A SERVQUAL-type instrument is most suitable for these methods
Note. A. Parasuraman. The SERVQUAL Model: Its Evolution And Current Status. (2000).
Paper presented at ARL Symposium on Measuring Service Quality, Washington, D.C.
Association of Research Libraries
http://www.arl.org/stats
Assessment can…
•
•
•
•
Support decision making
Establish accountability
Strengthen organizational mission, goals, and objectives
Realign library goals and mission statements with the
needs of users
• Identify trends in user behaviors and technology
• Measure learning outcomes and impact
• Improve programs and services
Association of Research Libraries
http://www.arl.org/stats
Use Assessment Data for Good!
Association of Research Libraries
http://www.arl.org/stats
Association of Research Libraries
http://www.arl.org/stats
What’s in a word?
quality
library
What makes a
?
“Quality much like beauty is in the eye of the beholder”
Association of Research Libraries
http://www.arl.org/stats
What’s in a “Library”?
A word is not crystal, transparent and
unchanged; it is the skin of a living
thought, and may vary greatly in color
and content according to the
circumstances and time in which it is
used.
--Justice Oliver Wendell Holmes
Association of Research Libraries
http://www.arl.org/stats
Interpreting Data
Association of Research Libraries
http://www.arl.org/stats