Transcript Assessment - Bringing Theory to Practice | Supporting and

Moving from Concept to
Meaningful Assessment:
Using Logic Models to Maximize
Efforts & Clarify Results
Ashley Finley, Ph.D
Senior Director of Assessment & Research, AAC&U
National Evaluator, Bringing Theory to Practice
Bringing Theory to Practice Well-Being Research Project Webinar
April 29, 2015
BTtoP 2015-2017 Grant Emphases
(http://www.bttop.org/sites/default/files/public/bttoprfp20152017.pdf)
O Category I: Well-Being Seminar Grants
O Up to $1,000)
O Provide support for campus-wide planning discussions (seminars) that give
focused attention to a particular dimension of the well-being of students,
faculty, or other leaders in creating and maintaining an engaged culture for
learning. Seminars should deepen participants’ understanding of the
institution’s commitment to whole-person development and the learningrelated policies and practices, both inside and outside the curriculum, that
support such endeavors
O For examples of other campus seminars, see: http://www.bttop.org/grantsfunding/category-1-seminar-mini-and-student-grants
O Category II: Well-Being Research Project Grants
O Up to $10,000, plus institutional matching)
O Implement and assess a campus-based two-year research project that
gives focused attention to the well-being of students, particularly those
traditionally underserved by higher education. Projects should, through the
gathering of evidence, provide justification for best practices for deepening
and sustaining the institution’s commitment to whole-person development
O http://www.bttop.org/grants-funding/category-2-program-start-or-researchinitiative-grants
Requirements for Well-Being Research Project Grants
Well-Being Research Project Grants proposals will be expected to
include the following three components and shouldn’t exceed 8 pgs:
O 1) Name and contact information of the primary contact person
(PI). Dates and timeline of implementation. Amount of funds requested
(budget requirements listed in RFP).
O 2) Proposal Narrative—organized in the following categories:
O Purpose of and rationale for the project
O Participants by campus area and rationale for their selection
O Facilitation process and proposed initial guiding questions that reflect
the context and culture of the institution
O Anticipated outcomes and subsequent action steps. A logic model
outline or its equivalent is expected.
O How the proposed work will link the research objectives to advancing
the larger campus strategy of attending to well-being as a core
purpose
O Evaluation and reporting information (requirements listed in RFP;
additional evaluation is always encouraged)
O 3) Budget (guidelines in RFP)
Questions?
Why Logic Models?
The problem with program assessment…
Curriculum
Assessment
Outcomes
Assessment Challenges
O What are the long-term impact goals (the vision
for change) vs. more immediate outcomes?
O How is what we propose to do actually connected
to the changes (outcomes) we propose to get?
O What types of evidence are needed to say
something meaningful about outcomes?
O How do we think holistically beyond grant
resources to other resources on campus that can
help facilitate change toward outcomes?
Moving from Goals to Expected Outcomes
Short-Term
EXPECT
to see
Intermediate
Would
LIKE to
see
Long-Term
Would
LOVE to
see
The need for collaboration necessarily increases as you move toward LOVE –
ability to affect outcomes requires greater resources and input
Resources needed to
start or keep going:
INPUTS
(What is
needed for the
process?)
ACTIVITIES
(What will
students be
asked to do?)
Actions needed to produce outputs:
OUTPUTS
(What counts
as good
evidence?)
OUTCOMES
(What should
improve as a
direct result
of efforts that
contribute to
the long-term
vision?)
Impact
Goals
(What is the
hope for the
future for
students,
for faculty,
for the
institution?)
Products
needed to
assess
outcomes,
“countables”:
Expected
Changes:
short, intermediate
Long-term
vision for
change
Resources needed to
start or keep going:
Actions needed to produce outputs:
INPUTS
(What is
needed for the
process?)
ACTIVITIES
(What will
students be
asked to do?)
•General
education
•Teaching and
learning center
•Counseling
center
•Student
affairs
•Student
orientation
•Career center
•Alumni center
•Group dialogue
•Small group
problem-solving
or applications
•Engagement
with
contemporary
issues, real-life
questions
OUTPUTS
•Individual/
group
Reflection
•Summary of
group
dialogue
•Critical
application of
well-being to
course
material
•Multimedia
Products
needed to
assess
outcomes:
OUTCOMES
•Flourishing
•Self-esteem
•Resilience
•Persistence
•Selfauthorship
•Satisfaction
with life
•Hope
[Resource:
University of
Pennsylvania
Positive
Psychology
Center]
Expected
Changes:
short,
intermediate:
Impact
•Build
culture of
well-being at
institution
•Increase
institutional
commitment
to whole
student
development
•Increase
commitment
to
“flourishing”
across
campus
community
Questions?
Basic Principles of Logic Models
O Should be developed collaboratively
O Intended to simplify plan for development and
function as communication tools
O Attentive to maturation of projects over time
O Start with outcomes, not inputs
O Assumptions underlie all dimensions
Selected Logic Model Readings & Resources:
• Grayson, Thomas. 2012. “Program Evaluation in Higher Education.” in Handbook
on Measurement, Assessment, and Evaluation in Higher Education. Charles
Secolsky & Brian Denison (Eds).
• Finley, Ashley. 2012. “Assessment and Evaluative Studies as Change Agents in
the Academy.” in Transforming Undergraduate Education. Donald Harward (Ed).
• Kellogg Foundation Logic Model Development Guide:
http://www.wkkf.org/resource-directory/resource/2006/02/wk-kelloggfoundation-logic-model-development-guide
A Case Study:
Dartmouth Mentoring with Purpose
O Mentoring With Purpose = collaborative project to foster a world
IMPACT
O
O
Activities
O
view of looking out for others; supporting the recently-launched
Dartmouth By-Stander Initiative (against sexual assault) by
engaging students as allies; and assessing the effects of cascade
mentoring on the campus, community, and participants.
A coordinated mentoring reflection program will allow students—
and faculty and staff members--to work in a "cascade fashion"
with people above them, beside them, and below them. Outcomes
Through an annual mentoring summit, dialogue groups,
classroom discussions, and their own mentoring of others,
students--with faculty, staff, community leaders--will learn more
about themselves; about those whom they mentor; & about the
possibilities & need for social change. They will address
significant life questions while mentoring, helping them to
flourish and to nurture others in flourishing as well.
Aim of the program is to involve 200 students in first mentoring
summit & 100 students in early discussion groups with faculty &
staff. Then expand to include larger numbers of students &
mentees and in surrounding communities, with eventual aim of
constituting a national civic engagement model. IMPACT
Defining Long-Term Impact, Intermediate, & Short-Term Outcomes:
Mentoring with Purpose Program
EXPECT to see
Engage students as
allies, to work in a
"cascade fashion" with
people above them,
beside them, and
below them
Would LIKE to see
• assess effects of
cascade
mentoring on
campus,
community, &
participants
• Expand [program]
to include larger
#s of students &
mentees & in
surrounding
communities
Would LOVE to see
Collaborative project
to foster a world view
of looking out for
others…[with]…
Eventual aim of
constituting a
national civic
engagement model
Who needs to be involved to reach goals of programmatic
expansion and national impact?
INPUTS
ACTIVITIES
• New pres. w/
strong
commitment
to experiential
educ.
• Critical # of
committed
fac., staff &
students
• Coord. efforts
of Tucker
Foundation,
Council on
Service &
Eng. & Dean
of College
• Existing
mentoring
programs
• Relevant
academic
courses (CBL)
• Far-reaching
awareness of
mindfulness
• Plan overall
program w/
stakeholders
• Glean learning
from guest
lecturers
• Plan summit
• Create
recruitment
training,
implementation,
debriefing,
assess. plan for
dialogue grps
• Train presenters
& facilitators
• Annual
mentoring
summit
• Annual
mentoring
dialogue grps.
• Assessment
• Write summary
report
OUTPUTS
• Survey data on
mentoring
programs’ success
vs. past success
• Pre/post survey on
participants’
understanding of
personal values &
goals
• Pre/post survey on
ability to connect
learning outside of
class
• # of students, fac.,
staff, CLs as
mentors
• # of cross-campus
& comm. Partner
collaborations
• # of younger
people/peers
mentored
• Media coverage of
mentoring
activities
Outcomes
Impact Goals
For all participants:
• articulate value of
mentoring & rel.
bet. mentoring
theory & practice
Students will:
• Dem. under. of CE
& social change
through mentoring
• Increased
awareness of
potential for
personal growth
• Dem. know. of
mentoring skills
• Apply curr.
Learning to exp,
outside classroom
• More committed to
create just society
Faculty/Staff/CL
• Dem. Increased
know. of student
lives outside
classroom
• Involved
students better
understand who
they are & who
they might
become
• Dartmouth is
known as a
“mentoring”
college
• More connection
bet. academic
theory &
learning outside
classroom
• Stronger
community on
campus across
faculty, staff &
comm. Members
• Increase #
students
committed to
helping create a
just society
Mapping Outcomes Beyond the Student Level
INPUTS
ACTIVITIES
OUTPUTS
Student
• Program
Student
• connection of well-
resources
• Inputs from
student affairs?
•Curricular inputs?
being & mentoring to
coursework
•Group work
• Activities in co-curr
• Reflection papers
• Summary pts from
group discussion
among mentors
• Collaborative work?
Faculty & Staff
Faculty & Staff
Faculty & Staff
• No. of faculty &
staff involved
• Faculty & staff
dev. resources
Institutional
• Current assess.
• Inst. Research
• Student Affairs
• Counseling
center
• Teaching Center
•Faculty & staff
orientation
• Faculty & staff
development
workshops
Institutional
• Assessment
workshops
• Communication
strategies to
promote
awareness
•Transparency
Student
• # of courses/progs
w/ best practices
• # of comm. contact
hrs./integration with
course material
• Faculty/staff dev. hrs
Institutional
• increase in
assign/courses
focused on wellbeing
• Posters/banners on
well-being
• recognition event
OUTCOMES
Student
• Increased
flourishing
•Increased selfesteem
•Resilience
Faculty & Staff
• Use of pedagogies to
support mentoring
• Awareness of Wellbeing
• Bldg. Communities of
Practice
Institutional
• Evidence of
connection of wellbeing & learning & CE
• Retention
• Campus awareness
Questions?
Summary Considerations for
Working with Logic Models
O Have you separated long-term impact goals from
intermediate and short-term outcomes
O What are the activities of the program? What is
happening to effect change in students? What about
faculty? At the institutional level?
O What are students doing as a result of their participation
that can be counted as evidence? What other sources of
evidence will be used to assess outcomes? Ditto at
faculty and institutional levels.
O What resources exist that contribute to project goals and
outcomes?
Additional Assessment Resources
O Logic Models: http://www.wkkf.org/resource-
O
O
O
O
directory/resource/2006/02/wk-kellogg-foundation-logicmodel-development-guide
Bringing Theory to Practice Toolkit Instrument & Overview
document (includes the flourishing scale):
http://www.aacu.org/bringing_theory/assessmenttools.cf
m#Tlkt
University of Pennsylvania Positive Psychology Center:
http://www.positivepsychology.org/ppquestionnaires.htm
AAC&U VALUE Rubrics (direct assessment of learning
outcomes, including civic engagement & intercultural
competence): http://www.aacu.org/value/index.cfm
Wabash Center for Inquiry in the Liberal Arts:
http://www.liberalarts.wabash.edu/study-instruments/
Post-Webinar Questions?
O Questions about grant guidelines or due
dates: [email protected]
O Questions about logic models or project
assessment: Ashley Finley, [email protected]