Transcript Mansfield Library - Montana State University

Kate Zoellner
Associate Professor
Assessment Coordinator
Sue Samson
Professor
Library Instruction Coordinator
1
LEARNING AND IMPACT
Outcome Metrics
“The consequences of an individual’s contact with the library.”
Learning Outcomes
“If students learn from their use of library resources and services… in terms of skills,
values, and attitudes.”
Impact Outcomes
“Concerned with user satisfaction, opinion, and perceptions, including satisfaction with use
of the library’s services and programs. They may also include the impact or the
effectiveness of library policies and management on services as well as the organizational
culture.”
Dugan, Hernon, & Nitecki, 2009
2
INFORMATION LITERACY LEARNING OUTCOMES
L I B R A RY I N S T R U C T I O N R U B R I C
B A S E D O N T H E A C R L
I N F O R M AT I ON L I T E R A CY
C O M P E T E N CY S T A N D A R D S F O R
H I G H E R E D U C A T I O N
1. The information literate student determines
the nature and extent of the information
needed.
2. The information literate student accesses
needed information effectively and
efficiently.
3. The information literate student evaluates
information and its sources critically and
incorporates selected information into his or
her knowledge base and value system.
4. The information literate student, individually
or as a member of a group, uses information
effectively to accomplish a specific purpose.
5. The information literate student understands
many of the economic, legal, and social
issues surrounding the use of information
and accesses and uses information ethically
and legally.
SCOPE OF THE LIBRARY
INSTRUCTION PROGRAM
• 522 Curriculum integrated
classes
• 10,665 Students
• 24,789 In-person reference
desk transactions
• 1,000 Virtual Reference
transactions
3
INSTRUCTIONAL ASSESSMENT MEASURES
Trend Data
•
Numbers
Curriculum Integrated
Classes
Reference Desk
Transactions
• READ (Reference Effort
Assessment Data)
• Chat Transcript Content
Analysis
• Online Student Feedback
• Online Faculty Feedback
• Learning Outcomes
• Peer review of teaching
In-office Consultations
• READ assessment
4
5
How might you apply what
you learned in this session
to your assignment?
In your research, what
might you do differently
based on what you learned
in this session?
6
7
8
LEARNING OUTCOMES ASSESSMENT
Example: During spring semester 2014, learning outcomes were analyzed
based on the library instruction rubric.
• Faculty identified the rubrics at each level that are incorporated into their
instruction for scheduled classes.
• Scheduled classes at each level were randomly selected to complete an
online set of learning outcomes which were then graded using a 4.0
scale.
• The Library Instruction Group made recommendations based on these
findings to refine and augment the library instruction program.
9
RECOMMENDATIONS FOR INSTRUCTION
Emphasize instruction on the following topics:
•
Copyright and plagiarism
• Develop a list of examples of why copyright matters to students
• Develop copyright modules
•
Free and fee-based information
•
Legal/economic impact on access to information
Survey departmental faculty to identify key concepts from our rubric they consider
most important.
Reconsider curriculum-integrated approach to information literacy instruction
especially in relationship to new ACRL Framework.
10
IMPACT OUTCOMES TOOLS
To capture library users’ satisfaction, opinions, and perceptions:
Research-Based
• Focus Groups
• Interviews
• Open Feedback
• Surveys
Additional
• Advisory Groups
• Liaison Communications
11
IMPACT OUTCOMES ASSESSMENT
Interviews
Surveys
 Faculty Research Practices
 ClimateQUAL®
 Content Analysis
 Website Use & Navigation
 Code and theme
Open Feedback
 Feedback Form
 Code and theme
 Compare with norms
 LibQUAL+®
 Content Analysis
 Compare with norms
 Trend across LQ survey years
 Review
 No Book
 Review
12
LIBQUAL+®
“A suite of services that libraries use to solicit, track, understand, and act upon
users'—Survey
opinions
of service quality.”
LibQUAL+®
Preview
This window will not appear on the live survey .
Conducted in spring 2003, fall 2006, and spring 2010; forthcoming spring 2015
Survey includes:
Welcome!
•
•
•
•
22 core items that measure
users’ perceptions of service quality
•
Affect of Service
•
Information Control
•
Library as Place
Questions on:
•
General Satisfaction
•
Information Literacy
•
Library Use
Demographic Information
Open-ended Comment Box
We are committed to improving your library services. Better understanding your expectations will help us tailor those services to your needs.
We are conducting this survey to measure library service quality and identify best practices through t he Association of Research Libraries' LibQUAL+® program.
Please answer all items. Thank you for your participation!
Please rate the following statements (1 is lowest, 9 is highest) by indicating:
Minimum -- the number that represents the minimum level of service that you would find acceptable
Desired -- the number that represents the level of service that you personally want
Perceived -- the number that represents the level of service that you believe our library currently provides
For each item, you must EITHER rate the item in all three columns OR identify the item as " N/A" (not applicable). Selecting " N/A" will override all other answers for that item.
My Minimum
Service Level Is
When it comes to...
My Desired
Service Level Is
Low
Perceived Service
Performance Is
High Low
High Low
High N/A
1) Employees who instill confidence in users
2) Making electronic resources accessible from
my home or office
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
3) Library space that inspires study and learning
4) Giving users individual attention
5) A library Web site enabling me to locate
information on my own
My Minimum
Service Level Is
When it comes to...
My Desired
Service Level Is
Low
6) An environment conducive to learning
through classes, programs, activities, and
meetings
Perceived Service
Performance Is
High Low
High Low
High N/A
1
2
3
4
Association of Research Libraries, 2014
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
7) Employees who are consistently courteous
8) The printed library materials I need for my
work
9) Quiet space for individual activities
10) Readiness to respond to users' questions
My Minimum
Service Level Is
When it comes to...
My Desired
Service Level Is
Low
Perceived Service
Performance Is
High Low
High Low
High N/A
11) The electronic information resources I need
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
1
2
3
4
5
6
7
8
9
N/A
12) Helpful online guides and tutorials
13) Employees who have the knowledge to
answer user questions
14) Library staff providing help that assists in
finding information needed now while improving
my research skills
13
6.43
6.33
6.31
6.14
5.96
6.08
6
5.82
6.01
6.1
5.89
6.02
6.08
5.71
5.68
5.74
5.56
5.14
5.26
5.5
5.41
5.46
5.57
5.6
5.74
5.84
6.08
6.22
6.23
6.33
6.43
6.48
6.58
6.71
6.55
7.13
6.99
6.59
6.44
6.5
6.51
6.63
6.63
6.77
6.79
6.91
6.91
6.94
6.93
6.86
6.7
6.51
6.67
6.35
6.1
6.07
6.14
6.18
6.33
6.18
6.28
5.95
6.09
5.83
5.67
5.75
5.67
6
5.92
5.89
6.06
6.25
6.12
6.19
6.3
6.29
Mean
6.5
6.28
6.35
6.3
6.54
6.75
6.71
6.62
7
6.87
6.9
6.91
7.02
7.07
7.25
7.14
7.29
7.22
7.31
7.5
6.53
7.6
7.75
6.74
Undergrad 03
Undergrad 06
Undergrad 10
Graduate 03
Graduate 06
Graduate 10
Faculty 03
Faculty 06
Faculty 10
Staff 03
Staff 06
Staff 10
All UM 03
All UM 06
All UM 10
All ARL 03
All ARL 06
All ARL 09
Information Literacy 2003, 2006, 2010 by User Group
5.25
5
The library helps me stay abreast of
developments in my field(s) of interest.
The library aids my advancement in my
academic discipline or work.
The library enables me to be more efficient The library helps me distinguish between The library provides me with the information
in my academic pursuits or work.
trustworthy and untrustworthy information
skills I need in my work or study.
Information Literacy Question
14
LIBQUAL+® FINDINGS
PERCEPTIONS
Areas of greatest concern (gap between
perceived and desired service levels):
•
•
•
•
A library Web site enabling me to locate information
on my own
Print and/or electronic journal collections I require
for my work
Ready access to computers/internet/software
(question not on previous surveys)
Making electronic resources accessible from my
home or office
Areas of greatest importance (desired service
levels):
•
•
•
Print and/or electronic journal collections I require
for my work
Making electronic resources accessible from my
home or office
A library Web site enabling me to locate information
on my own
G ENERAL S ATISFACTION
& AFFECT OF SERVICE
General satisfaction increased from 2003 to
2006 to 2010; trajectory parallels that of ARL
libraries.
Staff are the outlier group, their satisfaction
decreased slightly between 2006 and 2010 in
terms of their satisfaction with the way they
are treated at the library and the library’s
support for their learning, research, and
teaching needs.
Overall rank of Affect of Service:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Employees who are consistently courteous
Willingness to help users
Employees who deal with users in a caring fashion
Employees who have the knowledge to answer user
questions
Dependability in handling users' service problems
Readiness to respond to users' questions
Employees who understand the needs of their users
Employees who instill confidence in users
Giving users individual attention
15
CHANGES MADE BASED ON ALL ASSESSMENTS
Outreach




New employee orientation
Newsletter
Non-academic unit liaisons
Staff Social and workshops
Facilities
 Comfortable furniture
 Quiet study spaces
Web Site
 Discovery service
 LibGuides
Collections
 E-journals and Electronic
resources
 Digitization projects
 Popular reading materials
Services
Instruction
 Connections with liaison
librarians
 Workshop series




Chat reference
Exhibits and programming
Software and equipment
Supplies
16
CHALLENGES
LEARNING OUTCOMES
IMPACT OUTCOMES
• Partnerships with campus
faculty
• Capturing non-users
• Curriculum-integrated
information literacy outcomes
• Tracking individuals across their
academic careers or
employment at UM
• Access to student work
• No clear benchmarks
• Tracking students across their
academic careers
• Responding to results
17
BIBLIOGRAPHY
Association of College and Research Libraries (2000). Information literacy
competency standards for higher education. Chicago, IL: American Library
Association. Retrieved from
http://www.acrl.org/ala/mgrps/divs/acrl/standards/standards.pdf
Association of Research Libraries (2014). About LibQUAL+®. Retrieved from
https://www.libqual.org/
Dugan, R. E., Hernon, P., & Nitecki, D. A. (2009). Viewing library metrics from different
perspectives: Inputs, outputs, and outcomes. Santa Barbara, CA: Libraries
Unlimited.
18
MSU LIBRARY - THE BALANCED SCORECARD
The Balanced Scorecard (BSC) is a mechanism to gather assessment data and
drive strategic change in an organization. What is unique about the BSC, is that it
emphasizes balancing assessment criteria among four perspectives:
•
Financial information
•
Customer (i.e. student and faculty) requirements
•
Internal management processes
•
Innovation and learning
The premise is that a structured process to balance several criteria in different
categories can lead to better choices and more successful implementation than
over-reliance on any one set of factors. This entire process is driven by the
desired change-agenda as laid out in organization’s strategic plan.
19
PERSPECTIVES OF BSC PREDICATED ON
STRATEGY
Financial
What do our financial
stakeholders expect or
demand?
Customer
Who are our target
customers, what are
their expectations,
and what is our value
proposition in serving
them?
STRATEGY
Internal Process
At what business
processes must we
excel to drive value for
customers?
Employee Learning &
Growth
How do we align our
intangible assets to
improve our ability to
support our strategy?
20
BENEFITS OF BSC
•
Foster accountability.
•
Align employees with organizational goals.
•
Enhance resource allocation decisions.
•
Improve collaboration
•
Generate improved financial results.
AND … MOST CRITICALLY: Execute the strategy!
21
MSU NATIONAL LEADERSHIP GRANT AWARD
Problems:
1. Assessing visitation and use of the digital library is difficult because we lack
standards for web analytics, so reporting is full of inaccuracies
2. No comprehensive studies exist to prove that institutional repositories can have a
positive impact on author citation rates and, perhaps, university rankings
Proposals:
1. Develop standards to improve the accuracy of web analytics reporting
2. Develop an assessment framework to help evaluate the impact of institutional
repositories on author citation rates and university rankings
22