Transcript Customer Expectations” and

Outcomes Assessment
A Sustained Effort to Improve the
Quality of Student Learning
Four Perspectives
1. The institution and
the communities
served in the life of
the library
2. The library and
academic
departments and
programs in the life
of the institution
3. The library,
department/program,
and institution in the
life of the user/
customer
4. The library, program/
department, and
institution in the life
of stakeholders
Peter Hernon (October 2004)
2
Measurement Questions
1.
2.
3.
4.
5.
6.
7.
How much?
How many?
How economical?
How prompt?
How accurate?
How responsive?
How well?
8. How valuable?
9. How reliable?
10.How courteous?
11.How satisfied?
Peter Hernon (October 2004)
3
Examples of Quality Areas for Libraries
Instruction, teaching,
Collections
Fostering learning
Programs
Facilities, equipment
Services
Staff
Quality
(examples)
Management
Leadership
Peter Hernon (October 2004)
4
Measurement Indicators of Quality
Input
Customer-focused
Choices
Outcomes,
including impacts
Output
Peter Hernon (October 2004)
5
• “Assessment and evaluation are intended as means to
document educational quality and institutional
effectiveness, foster institutional improvement, and
demonstrate accountability Unfortunately, without a clear
sense of the purpose for assessment, knowledge of what is
to be assessed, and understanding how the results will be
used, assessment efforts, all too often, become an end in
themselves. If the results of assessment are not used to
information planning and decisions, college and
universities often find themselves in positions of being
data rich and information poor.” Ronald L. Baker, Outcomes
Assessment in Higher Education, p. 12
Peter Hernon (October 2004)
6
Outcome Assessment in Libraries
– User oriented rather than institution centered
• Statements about what students will know/think/be
able to do as a result of our programs
• Not statements about what the library should/could
do to bring about desired outcomes
– Measures changes in library users as a result of
their contact with an academic library’s
programs, resources and services
• What student knows, content, development of skills
and abilities, and acquire attitudes and values
Peter Hernon (October 2004)
7
Key Concepts
• Accountability
• Educational Quality
(continuous quality
improvement)
– Examples
• Cost
• Return on investment
• Student outcomes
– Examples
Peter Hernon (October 2004)
• Student learning
outcomes
• Research outcomes
8
Student Outcomes
•
Student outcomes refer to aggregate statistics on groups of students
Graduation rates
Transfer rates
Retention rates
Employment rates for a graduating class)
Such outcomes are institutional outcomes and are used to compare institutional
performance
These outcomes are outputs and reflect what the institution has accomplished
they do not reflect what (or how much) students learned
they do not measure changes in students themselves due to their college
experience
Peter Hernon (October 2004)
9
Research Outcomes
• Analyze a problem statement in its full complexity
• Identify and relate relevant literature to the
problem under investigation
• Select or develop a theoretical framework
appropriate to solving the problem
• Select an appropriate procedure (research design
and methodologies) that address study objectives,
research questions, and hypotheses
• Adopt appropriate indicators of reliability and
validity
Peter Hernon (October 2004)
10
Outcomes Assessment
• Often focuses on the changes (one related to
learning) that result from:
• Completion of a program of study, including a major or minor
• Completion of a course
For either of above, factor in contact with library
resources, programs, or services—the role of the
library in enhancing the educational experience
and learning
Peter Hernon (October 2004)
11
Levels of Outcomes
Institutional-level outcomes
Program (college) -level
outcomes
Course-based outcomes
Peter Hernon (October 2004)
12
Learning is Define as Encompassing
1.
2.
3.
4.
5.
6.
7.
Knowledge leading to understanding
Abilities
Habits of mind
Ways of knowing
Attitudes
Values
Other dispositions that an institution and its programs
and services assert they develop
Source: Assessing for Learning, by Peggy L. Maki (2004)
Peter Hernon (October 2004)
13
The Assessment Cycle Views
Planning in Two Contexts
Institutional
Mission
Vision
Values
Impact
Quality
improvement
See next slide
Peter Hernon (October 2004)
14
Assessment Cycle (adapted from Assessing
for Learning, by Maki, p. 5)
Gather evidence
Mission
Goals
Objectives
Interpret
evidence
Identify
Implement
outcomes
Review
outcomes
Peter Hernon (October 2004)
change
15
Understanding the Full Experience
(Matching with Mission)
Outcomes
Outputs
Student outcomes
Student learning
outcomes
Scholarly
output
Research outcomes
Etc.
Inputs as work load, resource
allocation, and grants received
Peter Hernon (October 2004)
16
Alternative Perspective
Conveyance of knowledge through research and scholarship
Student outcomes
Teaching effectiveness
Student learning outcomes
&
Research outcomes
Peter Hernon (October 2004)
17
Documenting Student Learning Outcomes
Course
Program
Peter Hernon (October 2004)
Lifelong
learning
18
Key Questions (Related to
Student Learning Outcomes)
How well do students
– Transfer and apply concepts, principles, ways of
knowing, and problem solving across their major
program of study?
– Integrate their core curriculum, general studies, or
liberal studies into their major program or field of
study?
– Develop understanding, behaviors, attitudes, values,
and dispositions that the institution asserts it
develops?
Source: Assessing for Learning, by Peggy L. Maki (2004)
Peter Hernon (October 2004)
19
Seven Questions
1.
2.
3.
4.
5.
What should students learn?
How well are they learning it?
What do students know that they did not know before?
What can they do that they could not do before?
What measures and procedures does the institution
utilize to determine that the answers to the above
questions?
6. To what extent does the institution offer evidence that
demonstrates its effectiveness to the public?
7. What does the institution plan to do with this evidence
to improve outcomes, including the quality of the
educational experience?
Peter Hernon (October 2004)
20
Some Focal Points
Communication skills: oral and written
Critical thinking
Information literacy (e.g., evaluation)
Problem solving
Quantitative reasoning ability
Setting up a good research study
Peter Hernon (October 2004)
21
Student Learning Outcomes*
See Information Literacy Competency Standards for Higher Education (ACRL,
2000)
• Higher Ordered
• Lower Ordered
– Problem solving
– Critical thinking
– Skills
• See Bloom’s Taxonomy of
Educational Objectives
Peter Hernon (October 2004)
22
Verb Choice for Outcomes
•
•
•
•
•
•
•
•
Analyze
Apply
Demonstrate
Develop
Differentiate
Identify
Recognizes
Synthesize
• Not
– Discusses
– Understand
– Etc.
– An Action Plan for
Outcomes Assessment in
Your Library (ALA, 2002),
p. 72
• Etc.
Peter Hernon (October 2004)
23
Examples
• Outcome: “Students will think logically and
critically in solving problems; be able to evaluate,
critique, and apply the thinking of others; and
reach an appropriate conclusion”
• Outcome: “Students will demonstrate leadership
abilities”
– Leadership abilities evidenced in learning contexts
include team building and shared decision-making,
strategic planning, advocacy, and consensus building
and collaboration
Peter Hernon (October 2004)
24
Information Literacy: An Area of Library
Focus
• Association of College & Research Libraries,
“Information Literacy Competency Standards for
Higher Education” (2000)
– Offers a basis to develop student learning and some
research outcomes
See also Middle States Commission and standardized
form being developed at Kings’ College (Terry Mech)
Peter Hernon (October 2004)
25
Information Literacy: Shared Responsibilities
for Learning
Information Literacy Skills
Responsibility
Know: Determines the
Starts with the faculty
member; reinforced by
librarians
nature and extent of
information needed
Access: Efficiently and
effectively accesses
information sources
Evaluate Sources:
Critically evaluates
information sources
The librarian usually
leaders, with faculty
support
The librarian may lead
initially; faculty make the
ultimate determination from
student’s work product or
performance
Peter Hernon (October 2004)
26
Information Literacy (continued)
Information Literacy Skills
Responsibility
Evaluate Content:
Faculty leads in classroom
or other course context;
students also may consult
librarians, external subject
experts, or peers
Critically evaluates
information content;
considers impact on
students’ prior knowledge,
value system, and future
direction in life
Use: Uses information
found to accomplish a
specific purpose
Faculty leads; librarians can
reinforce
Peter Hernon (October 2004)
27
Information Literacy (Continued)
Information Literacy
Skills
Responsibility
Ethically/Legally:
Faculty and librarians
jointly and continuously
Understands the economic,
legal, and social issues
surrounding the acquisition
and use of information
Source: Middle States Commission on Higher Education,
Developing Research & Communication Skills (2003), p. 23
Peter Hernon (October 2004)
28
Institutional Effectiveness and Improvement
Outcomes (Impacts)
Service Quality*
Inputs
Satisfaction
Analysis/
Review/
Evaluation
Outputs
*Not LibQUAL+
Peter Hernon (October 2004)
29
Methods of Evaluating Student Learning (Outcomes)
• Direct Methods
• Direct Methods
– Embedded course
assessment
(performance on
assignments, etc.;
minute papers
– Portfolio assessment
– Performance
(internships, practicum,
student teaching)
– Professional jurors or
evaluators
– Testing (standardized
or pre- and posttesting)
– Analysis of theses/
senior papers (content
analysis, interview, or
oral defense)
Peter Hernon (October 2004)
30
• Direct Methods
• Direct Methods
– Think-aloud protocol
– Directed conversations
– Capstone course/
activity
– Minute paper
– Videotape and
audiotape evaluation
Peter Hernon (October 2004)
31
Example (Kathleen Dunn, p. 28)
• Core Competency: “locate and retrieve information, in all of its various
formats using, when appropriate, technological tools.”
• Scenario: “You’ve been selected to lead a group of
students on a four-week trip to another country. In
preparation you need to find background information on
the country’s history, current events, language, local
customs, art, and music. How would you locate the
information? What type of information would you expect
to find?
• How can we assess the extent of success--to which scenario met and
change--over time?
Peter Hernon (October 2004)
32
Methods of Evaluating Student Learning (Outcomes)
• Indirect Methods
– Surveys (self-reports)
• Indirect Methods
– Curriculum and
syllabus evaluation
– Exit interviews
– External reviewers
– Observation
– Self-assessment
– (Library) database
statistics
Peter Hernon (October 2004)
33
Public learns to use technology
• Using database A (or search engine B),
construct a search strategy to locate
information that:
applies Boolean operators to narrow the scope
of the search
applies basic and advanced search protocol to
retrieve needed information
Peter Hernon (October 2004)
34
• Objective: to reduce the number of
retrievals (hits) per search on aggregate
and specialized databases
– Is this objective measurable? What types of
measures would you apply?
– Can this objective be taught? How?
– How would you measure if anything was
learned?
– What will the results tell you?
Peter Hernon (October 2004)
35
Some Relevant Readings
• Outcomes Assessment in Higher Education
(Libraries Unlimited, 2004)—companion volume
coming in January 2006
• An Action Plan for Outcomes Assessment in Your
Library (ALA, 2002)
• The Journal of Academic Librarianship (Jan.-Mar.
2002); (Nov. 2002)
• Assessing for Learning, by Peggy Maki (Stylus,
2004)
• Textbooks from Middle States Commission
Peter Hernon (October 2004)
36
Questions