Transcript Customer Expectations” and

Covering Assessment
in LIS Education and
in the Profession
Megan Oakleaf
Peter Hernon
Karin De Jager
Library Assessment Conference
August 2008
1
Panel Overview

Introduction
◦ Some examples of what LIS education is
doing
◦ Assessment
 Student outcomes
 Student learning outcomes
The Assessment Toolkit
 Assessment in Context
 A view from South Africa

Library Assessment Conference 2008
2
LIS Education
Status of using student learning outcomes
should guide programs
 Role of research at master’s level

◦ Research—Application of inquiry process
◦ Evaluation—Library centric
 Examination of program/service for
summative/formative evaluation
◦ Assessment—Connects libraries and broader
organizations to stakeholder expectations and
requirements
Library Assessment Conference 2008
3
Key Stakeholders behind Assessment

Government
◦ Federal
◦ State

Accreditation
◦ Regional Accrediting Organizations
◦ Program “Accreditors”
Library Assessment Conference 2008
4
Critical Issues

How do we build
“research” as a most
essential activity
within LIS education?

How do we build
research as a more
essential activity
among libraries and
librarians?
Library Assessment Conference 2008
5
Partnerships
LIS schools/programs
An institution
Cross-disciplinary partners
Library Assessment Conference 2008
6
Other Partnerships
Libraries within the US
LIS Education
CE
Workshops
Speaking
Scholarship
Key professional
associations
ISSUES: How to address research
and assessment?
Library Assessment Conference 2008
7
Assessment
1. Accountability: Meeting institutional mission—
effectiveness and institutional fiscal efficiency
AGGREGATE STATISTICS ON GROUPS OF
STUDENTS
GRADUATION RATES,
RETENTION RATES
TRANSFER RATES
EMPLOYMENT RATES FOR A
GRADUATING CLASS
Library Assessment Conference 2008
8
Assessment Cycle for Student
Learning Outcomes
Planning (Assessment Plan)
Gather “evidence”
Institutional
Mission
Vision
Values
Identify outcomes
evidence
Interpret
Review outcomes
Use the results
LibQUAL+ is
irrelevant
Library Assessment Conference 2008
9
Student Learning Outcomes--Examples
Conceptual
Leadership
Critical thinking
Problem solving
Information literacy
Global citizen
Values (moral, etc.)
Skills
Oral/written communication
Foreign language communication
Technological sophistication
Quantitative reasoning ability
Other
Library Assessment Conference 2008
10
Assessment
2.
Educational quality and improvement (e.g., student
learning)
WHAT SHOULD STUDENTS LEARN (IN A PROGRAM)?
HOW WELL ARE THEY LEARNING IT (IN THAT PROGRAM)?
WHAT MEASURES AND PROCEDURES DOES THE INSTITUTION
USE TO DETERMINE THAT IT IS EFFECTIVE?
TO WHAT EXTENT DOES THE INSTITUTION OFFER EVIDENCE
THAT DEMONSTRATES ITS EFFECTIVENESS TO THE PUBLIC?
WHAT DOES THE INSTITUTION PLAN TO DO WITH THIS
EVIDENCE TO IMPROVE OUTCOMES?
Library Assessment Conference 2008
11
Student learning outcomes exist at the
following levels
course
Program
Use of rubrics
Institutional
12
Direct Methods
Embedded course
assessment
(performance on
assignments, etc.;
minute paper)
 Portfolio assessment
 Performance
(internships,
practicum, student
teaching)
 Professional jurors or
evaluators
 Capstone
course/experience

Experimental research
designs), with pre- and
post-testing
 Use of standardized
tests
 Think-aloud protocol
 Directed conversation
 Videotape/audiotape
evaluation
 Analysis of
theses/dissertations/
senior papers (content
analysis, interviews, or
oral defense)

Peter Hernon
13
Indirect Methods
Surveys (selfreporting) and selfassessments
 Curriculum and
syllabus evaluation
 Exit interviews
 Observation


Peter Hernon
Other
14
Simmons’ MLIP Leadership Model

The curriculum and assessment activities are
guided by a leadership model, which was
adapted from a model developed by the
National Center for Healthcare Leadership. The
model consists of twenty-five distinct leadership
competencies in three broad areas:
Transformation, Accomplishment, and People.
http://web.simmons.edu/~phdml/docs/phdmlip_model
s.pdf
15
Assessment in Context

Learning in context is authentic & meaningful.
◦ Students apply skills as they would in the real world.

Learning in context is active.
◦ “Students construct meaning and knowledge: they do not
have meaning or knowledge handed to them in a book or
lecture. Learning, then, is a process of students ‘making
sense’ of how things fit together; factual and procedural
knowledge is built along the way” (Shavelson & Baxter,
1996).

Learning in context is open-ended &
acknowledges more that one right
approach/answer (Shepard, 1996).
Library Assessment Conference 2008
16
Example LIS Assignment
Planning, Marketing, & Assessing Library Services
Assignment Tasks:
 Locate a new or recently revised library
service & a host librarian
 For the service, develop:
◦ Project Management Plan
◦ Marketing Plan
◦ Assessment Plan

Present final plans to class & host librarian
Library Assessment Conference 2008
17
Library Service Examples
Virtual/IM reference
 Downloadable audio
 Gaming programs
 Single service points
 Information
commons
 Portals/blogs/wikis
 LibGuides

Digitization
 Orientations &
outreach
 Book clubs &
summer reading
programs
 Cafes/coffee bars

Library Assessment Conference 2008
18
Assessment Plan Outline











Service goals & link to strategic plan
Literature review
Service outcomes
Target audience
Methods & tools for evidence collection
Recommendations for pilot assessment
Analysis of evidence (data plan)
How assessors will know the outcome has been met
Result scenarios & decision making indicators
Recommendations for reporting
Responsible parties
Timeline
Library Assessment Conference 2008
19
Libraries Impacted (2007-2008)
Syracuse University Libraries
26 NYC-area public school libraries
Onondaga County Public Libraries (NY)
Lemoyne College Libraries
Wantagh Public Library (NY)
University of Rochester Libraries
SUNY ESF Libraries
Brooklyn Public Library (NY)
Mott Road Elementary School Library (NY)
SUNY Cortland Libraries
Jervis Public Library (NY)
Vogelson Public Library (NJ)
SUNY Brockport Libraries
Fayetteville Free Library (NY)
LeMoyne Elementary School Library (NY)
SUNY Binghamton Libraries
Cazenovia Public Library (NY)
Groton Elementary School Library (NY)
SUNY Upstate Medical University
Libraries
Cicero-North Syracuse HS Library
Paine Memorial Library (NY)
Fletcher Free Library (VT)
Drew University Libraries
Mid-York Library System (NY)
RIT Libraries
Broome County Public Library
Cazenovia Public Library (NY)
US Military Academy Library
New York University Libraries
Roanoke City Libraries (VA)
Celebration School Library (FL)
Green Mountain Library Consortium
Loyola University Libraries
Deschutes Public Libraries (OR)
Cornell University Libraries
Middlebury College Libraries (??)
Oneida-Herkimer BOCES School Library
System
University of Utah Libraries
University of New Hampshire Libraries
Schoharie Free Public Library (??)
Enoch Pratt Free Library (MD)
Regent University Libraries
University of Virginia Libraries
Northwestern University Libraries
Wake County Public Libraries (NC)
Boston College Libraries
Supreme Court Library (NV)
Whitesboro High School Library
Oneida Castle Elementary School Library
Rockefeller University Libraries
Norwood-Norfolk Central HS Library
Andrew J. Lanza Library (??)
UT – San Antonio Libraries
New England Law Library Consortium
Brandeis University Libraries
YouthBuild Charter School Library (PA)
George C. Marshall European Center for
Security Studies
Wellesley College Libraries
Library Assessment Conference 2008
20
Assessment Impact Examples

Librarians move forward on projects.
◦ Nearly all librarians say they’ll enact student plans, in
part or in whole.
◦ “We have paid thousands to ‘consultants’ who have produced
reports that don’t come anywhere near the level of detail and
professionalism that these students provided for us gratis. If we were
to move on this we could have a family-centered program at the
[children’s hospital] that would become a national model.”
–to hospital president and others from chair of pediatrics
◦ “If you were wondering if your project was ever touched – most
certainly! Your project has been the backbone of my knowledge and
launching point for inquiry. Hopefully in 2-3 months you will see
these items [downloadable audio] in the catalog and in our
marketing.”
--to student from Wake Public Libraries (NC)
Library Assessment Conference 2008
21
Assessment Impact Examples
Students gain professional positions.
 Student named Federal Library Technician of
the Year.
 Student recommended as chair of assessment
committee at New England Law Library
Consortium.

Library Assessment Conference 2008
22
Courses at selected
LIS Programs
Indiana
Texas
“Evaluating Programs and Services”
“Evaluation of Systems and Services”
“Outcome Based Evaluation of Programs and
Services”
“Evaluation of Library and Information Services
& Systems”
“Evaluation of Resources and Services”
“Administration”
Wisconsin
Hawaii
Florida State
ECU
“Information Services Management”
“Teaching Information Technology Literacy”
“Planning, Evaluation & Financial Management”
Theme & component throughout program
UIUC
Michigan
Rutgers
Library Assessment Conference 2008
23
From SA point of view - 2 implicit
assumptions:
The workplace requires evaluation &
assessment activities from librarians
 Library schools are teaching some of
the competencies required for these
activities
 Little evidence of either in local
practice

Library Assessment Conference 2008
24
Reasons
No standardized data collection required
from libraries
 Inevitable result: not a strong culture of
assessment evident on the SA library
scene
 If evaluation & assessment not a high
priority in libraries - almost self-evidently
not high priority in library schools either

Library Assessment Conference 2008
25
Library education in SA
Schools/Departments generally small &
threatened with closure
 Reduced in number during the last 10 years
from 18 to 12; also more closures in sight
 Some that remain have merged with other
disciplines in order to survive;
 Or evolved other survival strategies; e.g.
diversifying into adjacent areas like knowledge
or records management, media studies &
publishing

Library Assessment Conference 2008
26
Two kinds of qualifications

English speaking universities: mainly post-graduate
Diploma after Bachelor’s degree
◦ To ensure that students have some subject
specialization



Other universities: first degree in librarianship with
somewhat less emphasis on subject specialization
2 qualifications initially envisaged as equal (both
took 4 years to complete); but
Gradual emergence of 3 year qualification in
information studies - much less subject
specialization required
Library Assessment Conference 2008
27
Implications






Librarians rather technicist in orientation
Focus on the practicalities of obtaining, managing &
provision of resources
Frequently not enough subject expertise to be
regarded as equals by faculty
Tend to concentrate on undergraduate needs &
information literacy of very diverse & frequently
underprepared student body.
Library performance measurement may be
regarded with suspicion
Fear that own institution ‘might be shown up’ - of
lesser quality than others
Library Assessment Conference 2008
28
University of Cape Town
Postgraduate diploma: small course on
performance measurement & evaluation
 6 teaching periods:

◦ objectives of performance evaluation,
◦ approaches to measuring
◦ few informal case studies & examples of processes &
procedures
Eventual need for evaluation skills in workplaces
emphasized
 Self-study projects on e.g. measuring in ILL depts,
assessment of infolit competencies & information
needs; statistics for electronic resources & web
usability studies
Library Assessment Conference 2008

29
Yet growing demand for evidence of
quality
SA Council for HE mandates national
institutional quality audits
 Libraries to provide evidence of quality &
impact of services on teaching & research

◦ Some assistance from CHELSA

Considerable interest in PM7 in 2007
◦ Ca 70 librarians from SA (total of nearly 200)

Influence of LibQUAL+
◦ Though language & structure very difficult at
institutions where English not first language of
student body
Library Assessment Conference 2008
30
Also problem with research in SA
SA research output declined since 1990s
 Respected researchers generally pale & male &
about to retire
 Results of our LibQual evaluation (2005)

◦ Loud & clear; postgraduates & researchers not happy
with Library resources & services
◦ Both faculty & postgraduates (i.e. both current and
future researchers) rated all of Information Control
below minimum expectations

Serious & sustained interventions required to
support & enhance the research enterprise
Library Assessment Conference 2008
31
Novel intervention SA library
education
Ambitious Library project to support
researchers in the Library
 Consortium of 3 large academic libraries
funded by Carnegie
 Intended to catch up with what was not learnt
in library school
 Program for librarians: ‘total immersion’ into
research enterprise

◦ Monitoring & measuring ALL activities essential for
improvement
Library Assessment Conference 2008
32
No research support without evaluation
& measurement


Two-week “Academy” for 6 mid-career librarians from
each institution for 2 (or 3) years
Best possible researchers talking about their own
research
◦ Wide range of disciplines & from very different epistemologies
Each participant also to produce potentially publishable
research paper: with data collection, measurement or
assessment component
 Research involves finding out & counting & measuring to
understand what is really going on –
 Whether in libraries or elsewhere in the research
enterprise

Library Assessment Conference 2008
33
Bibliography

Shavelson, Richard J., and Gail P. Baxter. "Linking Assessment with Instruction." A
Handbook for Student Performance in an Era of Restructuring. Eds. Robert E. Blum
and Judith A. Arter. Alexandria, Virginia: Association for Supervision and Curriculum
Development, 1996. IV-7:1 - IV-7:6.

Shepard, Lorrie A. "Why We Need Better Assessments." A Handbook for Student
Performance Assessment in an Era of Restructuring. Eds. Robert E. Blum and Judith
A. Arter. Alexandria,Virginia: Association for Supervision and Curriculum
Development, 1996. I-2:2 - I-2:7.
Library Assessment Conference 2008
34