Moving from Concept to Meaningful Assessment:

Using Logic Models to Maximize Efforts & Clarify Results

Ashley Finley, Ph.D

Senior Director of Assessment & Research, AAC&U National Evaluator, Bringing Theory to Practice Bringing Theory to Practice Well-Being Initiative Assessment Webinar March 5, 2014

The problem with program assessment…

Assessment Curriculum Outcomes

Assessment Challenges

O O O O What are the long-term impact goals (the vision for change) vs. more immediate outcomes?

How is what we propose to

do

actually connected to the changes (outcomes) we propose to get?

What types of evidence are needed to say something meaningful about outcomes? How do we think holistically beyond grant resources to other resources on campus that can help facilitate change toward outcomes?

Defining Impact from Outcomes…

EXPECT to see Would LIKE to see Would LOVE to see

Programmatic control over outcomes decreases as move toward love…because you need people beyond the program

Resources needed to start or keep going: INPUTS

(What is needed for the process?)

ACTIVITIES

(What will students be asked to do?)

Actions needed to produce outputs: OUTPUTS

(What counts as good evidence?)

OUTCOMES

(What should improve as a direct result of efforts that contribute to the long-term vision?)

Impact Goals

(What is the hope for the future for students, for faculty, for the institution?)

Products needed to assess outcomes, “countables”: Expected Changes: short, inter mediate Long-term vision for change

Resources needed to start or keep going:

needed for the

• • • • • INPUTS

(What is process?)

General education Teaching and learning center Counseling center Student affairs Student orientation • Career center • Alumni center ACTIVITIES

(What will students be asked to do?)

• Group dialogue • Small group problem-solving or applications • Engagement with contemporary issues, real-life questions Actions needed to produce outputs: • OUTPUTS Individual/ group Reflection • Summary of group dialogue • Critical application of well-being to course material • Multimedia Products needed to assess outcomes: • • • • OUTCOMES Flourishing Self-esteem • Resilience • Persistence Self authorship Satisfaction with life • Hope [

Resource: University of Pennsylvania Positive Psychology Center

] Expected Changes: short, intermediate: Impact • Build culture of well-being at institution • Increase institutional commitment to whole student development • Increase commitment to “flourishing” across campus community

Logic Models: Basic Principles

O O O O O Should be developed collaboratively Intended to simplify plan for development and function as communication tools Attentive to maturation of projects over time Start with outcomes, not inputs Assumptions underlie all dimensions

Case Study: A BTtoP Project Campus

O O O The [community engagement program participants] are a trained group of students who work with professors and students on cultural sensitivity and inclusive language. Their work will help the college community become more aware of the politics of power, the root historical causes for social issues, and best practices for determining community needs in an unbiased manner. This awareness and attention to understanding the complex web of attitudes and linguistic choices that make people culturally sensitive, will begin to make us conscious of alternative choices in our engagement with the full diversity of our community. Our basic premise is that phobias and “isms”, are socially conditioned and constructed and can be unlearned and deconstructed…[R]ecognizing and challenging inappropriate discriminatory language is key. The work of the [program participants] and supporting faculty members will create a more whole campus community that is better prepared for wider engagement.

Moving from Goals to Expected Outcomes

EXPECT to see Expected changes through community engagement program participation that lead to… Would LIKE to see Would LOVE to see • • • College community becomes more aware of politics of power, root historical causes for social issues, best practices for determining community needs Consciousness of alternative choices in engagement w/ full diversity of community Create more whole campus community better prepared for wider engagement Programmatic control:

Who needs to be involved beyond & in coordination with the community engagement program?

INPUTS ACTIVITIES • Community engagement program • Faculty advisory board for program • Center for public service • Common book program • Comm. Engage. Diversity • Inclusion Excellence Program • 3 presentations or panels and related discussion • Program training, comm. tour, grp disc.

• Film Screening & Panel • Collaborative projects • Program training sessions and workshops OUTPUTS (None specified)

[What are student participants asked to do as result of training that encourages them to engage or synthesize learning to meet intended outcomes? What are students asked to do as a result of attending panels or presentations that encourages them to engage or synthesize learning to meet intended outcomes?]

Outcomes

Proposed Evaluation plan:

• Program Evaluations • Flourishing scale & civic attitudes scale to internship seminar students led by participants+ focus groups with seminar students (compare w/ control grps of students not working w/ participants) • NSSE & CIRP ( to look at community engagement & well-being ) Impact Goals • Unlearn hurtful language & unconscious oppression • Develop sensitivity to & awareness of unique history & culture of local comm.

• Gain particular knowledge & awareness of major “isms” and phobias: sexism, homophobia, religious bias, racism, xenophobia, classism, able ism, and ageism.

Mapping Outcomes Beyond the Student Level

INPUTS Student • • • Program components SA programs First-yr seminars Faculty • • No. of faculty allies Faculty dev. $ Institutional • • • Current assess.

Inst. Research • Student Affairs • Counseling center Teaching Center ACTIVITIES Student • connection of well being to coursework • • Group work Activities in co-curr Faculty • Faculty orientation • Faculty develop. Workshops Institutional • Assessment workshops • Marketing discussions OUTPUTS Student • Reflection papers • Summary pts from group discussion • Collaborative project Faculty • # of courses w/ best practices • • # of comm. contact hrs/integration with course material Faculty dev. hrs Institutional • increase in assign/courses focused on well being • • Posters/banners on well-being recognition event

OUTCOMES

Student • Increased flourishing • Increased self esteem • Resilience Faculty • Use of engaged pedagogies • Awareness of Well being • Bldg. Communities of Practice • • Institutional • Evidence of connection of well being & learning & CE Retention Campus awareness

IMPACT Activities O O O O

Case Study: Dartmouth Mentoring with Purpose

Mentoring With Purpose = collaborative project to foster a world view of looking out for others; supporting the recently-launched Dartmouth By-Stander Initiative (against sexual assault) by engaging students as allies; and assessing the effects of cascade mentoring on the campus, community, and participants. A coordinated mentoring reflection program will allow students— and faculty and staff members--to work in a "cascade fashion" with people above them, beside them, and below them. Through an annual mentoring summit, dialogue groups, classroom discussions, and their own mentoring of others, students--with faculty, staff, community leaders--will learn more about themselves; about those whom they mentor; & about the possibilities & need for social change. They will address significant life questions while mentoring, helping them to flourish and to nurture others in flourishing as well. Aim of the program is to involve 200 students in first mentoring summit & 100 students in early discussion groups with faculty & staff. Then expand to include larger numbers of students & mentees and in surrounding communities, with eventual aim of constituting a national civic engagement model. IMPACT

INPUTS ACTIVITIES • • New pres. w/ strong commitment to experiential educ.

• Critical # of committed fac., staff & students Coord. efforts of Tucker Foundation, Council on Service & Eng. & Dean of College • Existing mentoring programs • • • • Plan overall program w/ stakeholders Glean learning from guest lecturers • Plan summit • Create recruitment training, implementation, debriefing, assess. plan for dialogue grps Train presenters & facilitators Annual mentoring summit • Relevant academic courses (CBL) • Far-reaching awareness of mindfulness • Annual mentoring dialogue grps. • Assessment • Write summary report OUTPUTS • Survey data on mentoring programs’ success vs. past success • Pre/post survey on participants’ understanding of personal values & goals • Pre/post survey on ability to connect learning outside of class • # of students, fac., staff, CLs as mentors • # of cross-campus & comm. Partner collaborations • # of younger people/peers mentored • Media coverage of mentoring activities Outcomes

For all participants:

• articulate value of mentoring & rel. bet. mentoring theory & practice

Students will:

• Dem. under. of CE & social change through mentoring • Increased awareness of potential for personal growth • Dem. know. of mentoring skills • Apply curr. Learning to exp, outside classroom • More committed to create just society

Faculty/Staff/CL

• Dem. Increased know. of student lives outside classroom Impact Goals • Involved students better understand who they are & who they might become • Dartmouth is known as a “mentoring” college • More connection bet. academic theory & learning outside classroom • Stronger community on campus across faculty, staff & comm. Members • Increase # students committed to helping create a just society

Summary Considerations for Working with Logic Models

O O O O Have you separated long-term impact goals from intermediate and short-term outcomes What are the activities of the program? What is happening to effect change in students? What about faculty? At the institutional level?

What are students doing as a result of their participation that can be counted as evidence? What other sources of evidence will be used to assess outcomes? Ditto at faculty and institutional levels.

What resources exist that contribute to project goals and outcomes?

O O O O O

Additional Resources

Logic Models: http://www.wkkf.org/resource directory/resource/2006/02/wk-kellogg-foundation-logic model-development-guide Bringing Theory to Practice Toolkit Instrument & Overview document (includes the flourishing scale): http://www.aacu.org/bringing_theory/assessmenttools.cf

m#Tlkt University of Pennsylvania Positive Psychology Center: http://www.positivepsychology.org/ppquestionnaires.htm

AAC&U VALUE Rubrics (direct assessment of learning outcomes, including civic engagement & intercultural competence): http://www.aacu.org/value/index.cfm

Wabash Center for Inquiry in the Liberal Arts: http://www.liberalarts.wabash.edu/study-instruments/