Transcript Division of Student Affairs - AALHE | Association for the

Outcomes-Based Assessment and
Program Review for Academic Support Units
Irene Girton, Associate Dean of the College, Academic Affairs
Brian Harlan, Assistant. Dean for Academic Assessment & Planning
Celestina Castillo, Assistant. Director, Center for Community Based Learning
AALHE, June 2012
Outcomes of today’s session

Understanding the value of focusing assessment on
learning achieved through academic support
services

Developing a project plan for initiating an academicsupport assessment program at your institution
2
Why assess in the margins?
Academic support services significantly:



3
contribute to the total student learning experience
facilitate retention and overall student success
lead to overall satisfaction with the institution
Occidental’s ASAP Project (2008-2012)
Academic Support Assessment Program

Funded by the Teagle Foundation

Methodology modeled on academic assessment and
program review

Involved 14 academic support units from across
Academic and Student Affairs divisions
4
Academic Support at Occidental
Academic support at Occidental includes:

Professional and peer mentoring (tutoring)

Intellectual and individual life skills

Academic planning

Awards and fellowships, pre-professional, and career
advising

Experiential learning and research
5
Academic Tutoring
and Advising
Experiential Learning
and Research
Career and Preprofessional Advising
Academic and
Intellectual Life
Advising Center
International Programs
Office
National Awards Advising
Multicultural Summer
Institute
Center for Academic
Excellence
Undergraduate Research
Center
Pre-Health Advising
OxyEngage
Academic Mastery
Program
Center for Community
Based Learning
Career Development
Center
Office of Religious and
Spiritual Life
Scientific Scholars
Achievement Program
Keck Language and
Culture Studio
6
Discuss at your tables

List either the names or functions of units at your
institution that you would consider qualifying as
"academic support,” and check the division where
each unit reports.

Discuss with colleagues at your table the following
questions:




7
Where are most of your units housed?
What challenges and opportunities does your organizational
structure present?
Can units be grouped based on function?
Is anyone on your campus positioned to lead a cross-divisional
project?
Implementation Process
1.
2.
3.
4.
5.
6.
8
Identify project directors
Define project scope, timeline, deliverables, and
cost
Develop training materials
Group units into cohort learning communities
Train unit directors
Develop cohort-specific timeline and set
expectations
Project Expectations
Each unit was asked to undertake the following:
9

Develop a mission statement

Develop student learning goals and outcomes (distinct from
operational goals and outcomes)

Develop a service-outcomes alignment map

Perform outcomes-based direct and/or indirect assessment

Develop a self study based on assessment

Hold a site visit

Develop a response and action plan based on external
report
Example: CCBL Assessment Plan
Center for Community Based Learning Mission – 1

The mission of the Center for Community Based Learning is to
institutionalize civic engagement in order to enrich student
learning and commitment to social responsibility, enhance
student-faculty relationships, and to make tangible
contributions toward solving social-economic problems
through reciprocal community partnerships.

By employing a unique civic engagement model created by
the CCBL for developing a range of classes, projects and
programs, the Center aims to build long-term partnerships
and sustainable projects with long-lasting impact in the Los
Angeles and Occidental communities.
10
Example: CCBL Assessment Plan
Center for Community Based Learning Mission -- 2
Guiding Principles
In alignment with our mission, and within the context of the four cornerstones
of the College mission (Excellence, Equity, Community, and Service),
CCBL:

utilizes a model of civic engagement that is shaped and owned by all
participating stakeholders, including students, faculty, and community
partners, to ensure long-term sustainability and reciprocity

addresses genuine needs and interests as defined by community
partners, faculty and students participating in specific partnerships

values community expertise as essential by involving community
partners in the education of students, and by encouraging them to
articulate desired outcomes and potential strategies
11
Example: CCBL Assessment Plan

Student Learning Goal 1: To develop self awareness, critical
thinking, understanding of others, and commitment to civic
engagement

Student Learning Outcome 1.1: Students will demonstrate increased sensitivity and
commitment to social responsibility in their written and/or oral community-based
project reports.

Student Learning Outcome 1.3: Students will demonstrate their critical understanding
of social issues with respect to specific course material through papers
and/or assignments.

Student Learning Outcome 1.4: Students who participate in CCBL sponsored workshops with
faculty and community partners, will collaborate to develop course syllabi and
community projects with the interests, skills, and resources of all involved in
the partnership.
12
Example: CCBL Assessment Plan

Faculty Learning Goal 1: To develop faculty expertise in
community-based learning by collaborating with local, regional,
national, and international partners for the mutually beneficial
exchange of knowledge, skills, and resources

Faculty Learning Outcome 1.1: Faculty who participate in CCBL sponsored workshops with
community partners and students, will collaborate to develop course syllabi and
community projects with the interests, skills, and resources of all involved in
the partnership

Faculty Learning Outcome 1.2: Faculty who are new to the college, and/or who have never
integrated CBL into their teaching, will establish community-based learning components for
courses across a wide range of disciplines.

Faculty Learning Outcome 1.3: Faculty who are members of the CCBL Faculty Advisory
Committee, along with CCBL staff, will share their CBL knowledge and expertise with
other faculty throughout the campus who express an interest in integrating CBL in their
courses.

Faculty Learning Outcome 1.4: Faculty will design CBL courses with clearly articulated
connections to CBL pedagogies; will include guided reflection aimed at integrating
academic and civic learning; and will grade students for their academic learning in connection to
their community projects.
13
Example: CCBL Assessment Plan

Operational Goal 1: To create opportunities for community based learning throughout the
curriculum and co-curriculum

Operational Outcome 1.1: Students will enroll in community based learning classes, CBL
internships, and independent courses throughout the disciplines.

Operational Outcome 1.2: Faculty, students, and community partners will participate in CBL
workshops and other events, aimed at CBL skill building.
Operational Goal 2: To increase the number of community partnerships by collaborating
with local, regional, national, and international partners for the mutually beneficial exchange
of knowledge, skills, and resources.


Operational Outcome 2.1: Community partners will participate in CCBL sponsored
workshops with faculty and students where space and structure will be provided for course
syllabi and community projects to be developed with the interests, skills, and resources of all
involved in the partnership.

Operational Outcome 2.2: CCBL staff will assist faculty and students in identifying
community partnerships appropriate for the objectives of CBL classes or co-curricular
community projects.

Operational Outcome 2.3: CCBL staff will coordinate efforts with other campus
departments such as ORSL (Office of Religious and Spiritual Life), IPO (International
Programs Office), ICC (Intercultural Community Center), Student Life, and Community and
Government, and Neighborhood Partnerships, to identify and cultivate potential new community
partners locally, nationally, and internationally.
14
Example: CCBL Assessment Plan
Alignment map: CCBL activities and outcomes
Learning
Outcomes
SLO 1.1
SLO 1.3
SLO 1.4
FLO 1.1
FLO 1.2
FLO 1.3
Classes and
EIA
projects
facilitators
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
P
OO 1.2
P
OO 2.1
OO 2.2
OO 2.3
15
Partnerships
Collaborations
Workshops
Faculty advisory
with community
with CDC, URC,
and events
committee
organizations
IPO, etc.
P
P
P
FLO 1.4
OO 1.1
Individual and
group
development of
course syllabus
P
P
P
P
P
P
P
P
P
P
P
P
P
P
Assessment Types
Direct Assessment
Reflective Essays
Presentations
Course Assignments
Proposals and Personal
Statements
16
Indirect Assessment
National Surveys (CIRP,
NSSE, etc.)
Departmental Student and
Alumni Surveys
Faculty Surveys
Student Focus Groups
Example: URC Direct Assessment
Rating Scale
Quality of Content
Unacceptable
Needs Improvement
Developing
Sound research that
Research faulty, ideas not Original ideas, but
contributes to the
original and not relevant research is faulty and not
scholarship of the
to the scholarship of the related to the scholarship
discipline, but ideas do
discipline
of the discipline
not appear very original
Exceptional
Original ideas result from
sound research that
contributes to the
scholarship of the
discipline
Research process is
clear, results are
Research process and
summarized with
results are described with
concrete, relevant
good examples, but
examples & conclusion is
conclusion is not clear.
clear and concise.
Organization &
Clarity
Process and results not
presented clearly,
examples seem irrelevant
and conclusion is not
clear.
Research process is
unclear, results are
described, but examples
are lacking, and
conclusion is reasonable.
Presentation Style
Presenter is unclear and
unenthusiastic, does not
use technology when
needed and cannot
respond to questions.
Presenter is articulate,
Presenter is uncertain but Presenter is articulate,
enthusiastic, uses
enthusiastic, does not use enthusiastic, uses
technology as appropriate
technology appropriately, technology as appropriate
and responds to
responds to questions
but responds to questions
questions knowledgeably.
hesitantly and unclearly. hesitantly and unclearly.
Comments:
17
Example: URC Direct Assessment
18
Example: CAE Indirect Assessment
What was the primary reason that brought you to
the CAE?
2.5%
help with writing
27.5%
help with math
70.0%
help with another
subject
19
Example: CAE Indirect Assessment
Overall, working with a writing advisor
improved my ability in academic writing.
Agree or
strongly agree
CAE writing advisors consistently offer
appropriate support that fits my needs.
My experience working with a writing advisor
helped me to use specific examples and
details appropriately and effectively.
My experience working with a writing advisor
helped me to improve my control of words,
sentences, and/or paragraphs.
My experience working with a writing advisor
helped me to better organize my thoughts
and present them more clearly.
My experience working with a writing advisor
gave me confidence in my ability to handle
most writing assignments.
Neither agree
or disagree
Disagree or
strongly
disagree
0%
20
50%
100%
Discuss at your tables


On your list of Academic Support units, think of what
types of assessment could be done (direct or indirect)
and indicate this next to the unit’s name.
Discuss with colleagues at your table the following
questions:
 How much direct assessment would be possible?
 Is there potential for direct assessment at your
institution that we haven’t already mentioned?
 Are there ways to create new student products that
would allow direct assessment?
21
Self-Study Template










Mission, learning goals and outcomes
Learning outcome assessment focus
Meeting needs of students
Program staff
Mission and diversity
Cultural competence
Collaboration and referral
Technology
Physical facilities
Looking to the future
22
External Review Summary Sheet
Program:
Date of Review:
Reviewer(s):
Instructions: This summary sheet will assist you with identifying key areas (strengths
and improvements needed) to address in your final report. Please complete this
summary within one month of your site visit and email to [email protected].
Please rate the following program review criteria using the following scale:
E = Exemplary
S=Satisfactory
N=Needs Improvement
U=Unclear/need more information
1. Mission and goals
1.1
The program’s mission effectively conveys the program’s purpose(s).
1.2
Program goals outline an effective strategy to accomplish its mission and facilitate student
learning.
1.3
Program goals are assessed systematically and results are shared among colleagues and
with participants.
1.4
Do you recommend any changes to the program’s mission and/or goals?
If so, please explain and advise.
23
N/A
Evaluation
E, S, N, U,
N/A
Post-Review Follow Up
Academic Support Unit Name
External evaluator report received.
Unit director responds to report in writing to Assoc. Dean and the appropriate
VP, with particular focus on the report's major findings.
Unit director and Assoc. Dean for Academic Affairs formulate action plan and
five-year assessment plan. These, with evaluator report, are sent to
appropriate VP
Unit director and Assoc. Dean for Academic Affairs meet with appropriate VP
to discuss, revise if needed, and commit to action plan.
Unit director implements action plan.
One year after site visit, assessment plan is activated and 5-year review
cycle is initiated.
24
Completed?
Action Plan Template
Item
Student learning outcomes
Student needs
Staff , space, technology,
budget
Mission and diversity
Collaboration
General
25
Owner
Steps
Target
Date
Completed
Date
Action Plan Common Themes

Reevaluate all stated SLOs; make changes as needed

Develop service map and align SLOs

Establish a process for assessing student needs by developing a student survey (or
reëvaluating any current surveys) in preparation for your next program review

Update and maintain website to current campus standards

Devise an outreach program to cultural clubs and other student organizations as appropriate

Meet at least monthly as a group with leadership of [specific other Academic Support
groups]; other units as appropriate

Collect student user data for all student contact – name, class year, student ID, specific use
of unit

Annual report including student user data and assessment data for 1-2 SLOs

Meet regularly with supervisor
26
Questions?
Irene Montefiore Girton
Associate Dean, Academic Affairs
[email protected]
Brian Harlan
Assistant Dean, Academic Assessment & Planning
[email protected]
Celestina Castillo
Assistant Director, Center for Community Based Learning
[email protected]
27