Transcript Developing High Quality Service

Effective Methods for
Assessing the Impact of Service-Learning on
Students, Institutions, and Communities
ANDREW FURC0
University of Minnesota
February 27, 2009
Problems with Service-Learning Evaluation
1.
2.
3.
4.
5.
6.
Program goals are unrealistic.
Expected outcomes cannot be easily measured.
Evaluation is an after thought.
Evaluation is not aligned to program goals
We claim impacts when we only measured outcomes.
The time frame is not commensurate for impacts to
manifest.
7. We view all service-learning experiences to be the
same .
8. We make bold claims after one study.
General Tips for Successful
Evaluation
 Set measurable, realistic goals
 Focus the evaluation
 Have your evaluation design match evaluation question(s)
 Assess the quality of the service-learning practice
 Secure an appropriate sample
 Have a realistic timeline for data collection
 Apply appropriate instruments & data collection techniques
 Use a systematic approach for data collection
 Have a plan for how the data will be analyzed systematically
Service-Learning Quality Matters




Principles (e.g., Wingspread Principles)
Balance Beam
Quadrant
Essential Elements
Service-Learning: A Balanced
Approach to Experiential Education
FOCUS
Service
Learning
PRIMARY INTENDED BENEFICIARY
Recipient
Provider
Service-Learning
Community Service
Volunteerism
Field Education
Internships
Assessing Service-Learning Quality
High Service
II
I
Unrelated
Learning
III
Adapted from the Service
Learning Quadrant,
Service Learning 2000
Center
Integrated
Learning
IV
Low Service
ESSENTIAL ELEMENTS OF
SERVICE-LEARNING
CLUSTER 1: LEARNING
1) CLEAR EDUCATIONAL GOALS
2) CHALLENGING EDUCATIONAL TASKS:
3) ONGOING ACADEMIC ASSESSMENT
CLUSTER 2: SERVICE
4) MEANINGFUL SERVICE TASKS
5) EVALUATION OF SERVICE OUTCOMES
From Essential Elements of Service-Learning. National Youth Leadership Council. Minneapolis, MN.
Essential Elements
of Service-Learning
CLUSTER 3: CRITICAL COMPONENTS THAT SUPPORT
SERVICE AND LEARNING
6) STUDENT VOICE
7) DIVERSITY OF PERSPECTIVES
8) OPEN AND EFFECTIVE COMMUNICATION
9) STUDENT PREPARATION
10) ONGOING AND CRITICAL REFLECTION
11) CELEBRATION
.
Assessing Student Outcomes
DOMAIN
OUTCOMES
ACADEMIC
Increased retention; content knowledge
and skills; higher order thinking
CIVIC
Civic responsibility; commitment to
service
CAREER
Career awareness and skills
PERSONAL &
SOCIAL
Self-esteem, empowerment, motivation,
engagement (academic, civic, social);
prosocial behaviors
ETHICAL/MORAL Values development
MEASUREMENTS FOR STUDENT OUTCOMES
1) CART (http://cart.rmcdenver.com/)
Bringle, Phillips, and
Hudson (2004):
Contains a series of validated
scales used in service-learning
studies.
MEASUREMENTS FOR STUDENT OUTCOMES
1) CART (http://cart.rmcdenver.com/)
2) Bringle, Phillips, and Hudson (2004)
3) Higher Education S-L Survey
(Civic Responsibility, Academic Attitude, Career
Development, and Empowerment) (Furco, 2000)
4) CASQ (Civic Action, Social Justice, Appreciation
of Diversity) (Moely et. al., 2002)
ISSUES IN S-L STUDENT ASSESSMENT
• Student service preferences matter
Individual Preferences for Service
TYPE
EXAMPLE
Charity
Serve food to the homeless on Saturdays
Empowerment
Service
Educate the homeless about social services available to
them
Public Work
Facilitate the opening of a homeless shelter
Participatory
Democracy
Work to secure legislation and citizen support that will
secure rights for persons.
Social Action
Students organize a camp out on campus to raise
awareness about homeless
Social Change
Work to reduce the number of homeless persons; train
homeless persons for jobs
Social Justice
Secure legal assistance for a homeless person who
was denied health services
ISSUES IN S-L STUDENT ASSESSMENT
• Student service preferences matter
• Service-learning extends student
learning
Aspects of Student Learning in
Service-Learning
Civic
Responsibility
Community-based
Learning
Experiences
Academic
Achievement
Connecting Learning
and Service
through Reflection
…Service
LEARNING ABOUT
SERVICE
Classroom-based
Learning
Experiences
…Learning
LEARNING ABOUT
THE SOCIAL ISSUE
LEARNING THE
COURSE CONTENT
Learning Aspects of Service-Learning
Plant Biology Edible Garden for the Homeless
Learning the
Course Content
Students learn the process of photosynthesis, the
development of nutrients in plants, and the life cycles of
various types of plants.
.
Learning about
Service
Students learn: to cultivate various edible plants;
the nurturing that different types of plants require;
assessment of plant quality; and garden care.
Learning about
the Social Issue
Students learn: why and how individuals become
homeless; the ways in which the homeless can
benefit from being served the food from the
gardent; the nature of homeless services in the
local area, etc.
Assessing Institutional Outcomes
1) Institutionalization
2) Institutional climate (e.g. sense of
community)
3) Learning communities
4) Faculty engagement in servicelearning
5) Other
Instruments to Measure Institutionalization
(Engagement and Service-Learning)
1)
2)
3)
4)
5)
7)
8)
Kellogg Forum Checklist
Committee on Institutional Collaboration
Benchmarks
Furco Rubric for Institutionalizing Service-Learning
Holland Matrix on Relevance to Mission
Campus Compact Indicators of Engagement
Bringle et al.’s Comprehensive Assessment for the
Scholarship of Engagement (CASE)
Carnegie Elective Classification for Community
Engagement
KELLOGG FORUM:
DIMENSIONS OF ENGAGEMENT
A CHECKLIST of ten principles:
1) Access to Learning
2) Enhanced Diversity
3) Civic Leadership
4) Public Scholarship
5) Social Well-Being
6) Trusted Voice
7) Public Spaces
8) Community Partnerships
9) Self Governance
10) Public Accountability
CIC BENCHMARKS OF ENGAGEMENT
7 BENCHMARKS that show evidence of:
1) Institutional commitment to engagement
2) Institutional resource commitments
3) Student Involvement in engagement
activities
4) Faculty and staff partnerships with
community
5) Institutional engagement with community
6) Assessing impact & outcomes
7) Resource/Revenue opportunities generated
Committee on Institutional Cooperation
CIC BENCHMARKS OF ENGAGEMENT
1. Evidence of Institutional Commitment to Engagement
1.1. The institution’s commitment is reflected throughout its administrative structure.
1.2. The institution’s commitment is reflected in its reward structure for faculty and staff.
1.3. The institution’s commitment is reflected in its policies and procedures designed to
facilitate outreach and engagement activities.
1.4. The institution’s commitment is reflected in its policies and procedures that are
responsive to students.
2. Evidence of Institutional Resource Commitments to Engagement
2.1. The institution shows evidence of leadership for engagement and outreach
activities.
2.2. The institution shows evidence of financial support for engagement through its
budgetary process.
2.3. The institution shows evidence that faculty and staff time is devoted to outreach
and engagement activities.
2.4. The institution includes engagement activities as part of its programs for faculty,
student and staff development.
CAMPUS COMPACT INDICATORS OF
ENGAGEMENT
INDICATORS that suggest the presence of engagement
at the institution:













Mission and Purpose
Academic and Administrative Leadership
Disciplines, Departments, and Interdisciplinary work
Teaching and Learning
Faculty Development
Faculty Roles and Rewards
Support Structures and Resources
Internal Budget & Resource Allocations
Community Voice External
Resource Allocation
Coordination of Community-Based Activities
Forums for Fostering Public Dialogue
Student Voice
OPERATIONALIZATION OF INDICATORS OF
ENGAGEMENT
A. Mission and Purpose
• The institution’s mission statement explicitly articulates its
commitment to the public purposes of higher education and
is deliberate about educating students for lifelong
participation in their communities.
• This aspect of the mission is openly valued and is explicitly
used to promote and to explain the civic engagement and
community building activities on and off campus.
• The institution demonstrates a genuine willingness to review,
discuss, and strengthen its commitment to civic engagement
and community building.
• All members of the campus community demonstrate their
familiarity with and ownership of the institution’s mission.
From Burack and Saltmarsh (2007), Advancing Civic Engagement through Strategic Assessment. NERCHE
FURCO SELF-ASSESSMENT RUBRIC FOR
SERVICE-LEARNING INSTITUTIONALIZATION
A RUBRIC built on 22 components that are
organized within five dimensions:
1) Philosophy and mission of service-learning
2) Faculty support and involvement in service-
learning
3) Student support for and involvement in servicelearning
4) Community participation and partnerships
5) Institutional support for service-learning
Three stages: Critical mass building; Quality building; Sustained
institutionalization
SELF-ASSESSMENT RUBRIC FOR THE INSTITUTION ALIZATION OF
SERVICE-LEARNING IN HIGHER EDUCATION (Revised 2003)
DIMENS IO N I: PHILOSOPHY AND MIS SIO N OF SERVIC E-LEARNING
A primary componentof service-learning inst itutonalization
i
is t he developmentof a campus-wide definit ion forservice-learning t hat
provides meaning,focus, and emphasisfor t he service-learning effort. How narrowly or broadly service-learning is defined on your
campus will effect which campus const it uent spart icipat e/donot part icipat e,which campus unit s will provide financialresources and
ot her support
, and t he degree t o which service-learning will become part of t he campus’ inst it ut ional fabric.
DIRECTIONS: For each of the four categories(rows), place a circle aroundthe cell that best represents the CURRENT status of the
developm ent of a definitio
n, philosophy,and m ission of service-learning.
DEFINITION OF
SERVICELEAR NING
STRATEG IC
PLANNING
ALIGNMENT
WITH
INSTITUTIONAL
MISSION
ALIGNMENT
WITH
EDUCATIONAL
REFORM
EFFORTS
STAGE ONE
Critical Mass Building
There is no campus-wide definition
for service-learning. The term
"service-learning" is used
inconsistently to describe a variety
of experiential and service
activities.
The campus does not have an
official strategic plan for advancing
service-learning on campus.
W hile service-learning
complements many aspects of the
institution's mission, it remains on
the peripheryof the campus.
Service-learning is rarely included
in larger efforts that focus on the
core mission of the institution.
Service-learning stands alone and is
not tied to other important, high
profile effort s on campus (e.g.,
campus/community partnership
efforts, establishment of learning
communities, improvement of
undergraduate teaching, w riting
excellence emphasis, etc.)
STAGE TWO
Quality Building
There is an operationalized
definition for service-learningon
the campus, but there is some
variance and inconsistency in the
application of the term.
Althoughcertain short-range and
long-range goals for servicelearninghave been defined for the
campus, these goals have not been
forma lized into an official strategic
plan that will guide the
implementation of these goals.
Service-learning is often mentioned
as a primary or important part of
the institution's mission, but
service-learning is not included in
the campus' official mission or
strategic plan.
Service-learning is tied loosely or
informally to other important, high
profile effort s on campus (e.g.,
campus/community partnership
efforts, establishment of learning
communities, improvement of
undergraduate teaching, w riting
excellence emphasis, etc.)
STAGE THREE
Sustained Institutionalization
The institution has a forma l,
universally accepted definition for
high quality service-learning that is
used consistently to operationalize
many or most aspects of servicelearningon campus.
The campus has developed anofficial
strategic plan for advancing servicelearningon campus, which includes
viable short-range and long-range
institutionalization goals.
Service-learning is part of the primary
concern ofthe institution. Servicelearningis included inthe campus'
official mission and/orstrategic plan.
Service-learning is tied formally and
purposefullyto other important, high
profile effort s on campus (e.g.,
campus/community partnership
efforts, establishment of learning
communities, improvement of
undergraduate teaching, w riting
excellence emphasis, etc.)
NOTES
FINDINGS AND LESSONS LEARNED
Based on data from 209 Colleges and Universities:
 =1.6, s.d. = .21
Overall Yr. 3 institutionalization level:  = 2.1, s.d. = .31
1) Overall Yr. 1 institutionalization level:
Year 1-3 change:  = .50, p = 0.018
2) Estimated 3-5 years to move from one stage to next
3) Internal assessments are more positive than external
assessments 3 out of 4 cases (n = 43) in Year 1.
Internal assessments are more negative than external
assessments 3 out of 4 cases (n = 42) in Year 3.
FINDINGS AND LESSONS LEARNED
4) The self-assessment process is more important
than the rubric score
5) The purposeful, ambiguous wording of the rubric
encouraged discussion and dialogue
6) The components of the rubric are interdependent
7) All components do not have to be at Stage Three to
achieve institutionalization
8) As you progress you may regress
ASSESSING COMMUNITY OUTCOMES
STRATEGIES FOR EVALUATING
COMMUNITY OUTCOMES
 Involve community partners/members in the




design of the evaluation
Narrow the impact area to isolate servicelearning’s effect
Secure pre-post measures; benchmark change
Support personal stories with data
Use pictures to support the data