Transcript California Community Colleges Registrars and Admissions Officers (CACCRAO) Development of Meaningful Learning Outcomes Dotti Cordell BS, MPH, BSN Director of Student Health Services San Diego City College May.

California Community Colleges Registrars
and Admissions Officers (CACCRAO)
Development of Meaningful Learning
Outcomes
Dotti Cordell BS, MPH, BSN
Director of Student Health
Services
San Diego City College
May 4, 2009
A Challenge to ...
rekindle
recapture
review
relearn
REEXAMINE
recharge
REMEMBER
replenish
RECONNECT
rechallenge
rejuvenate
recall
revisit
restore
REDISCOVER
recommit
reorient
redirect
And Restart?...
Presentation Objectives
 Review the process of
Outcomes Assessment
 Clarify misconceptions
relating to outcomes in
Student/Administrative
Services
 Explain the Six Column Model
utilized at SD City College
 Equip you with an
organized systematic
knowledge base in this
sometimes unfamiliar or
uncomfortable territory
 Outline potential trouble
spots
Choice and
Attitude
 I hope that you can turn what might be a
negative external mandate into a
positive internal challenge.
 I challenge you to view assessment as an
opportunity for admission program
improvement and a means to seek the
best possible way to serve students.
If you do what you've always done,
you'll get what you've always
gotten
Do You Feel A Time-Crunch?
 What do I need to know?
 What does my department
need to know?
 What do I do?
Don’t Panic
 You and your department are probably
already doing it.


You just may not know it.
Or you may not be documenting the process.
 Or you think there is a perfect way to do it
and you are afraid to jump in.
 Or it does not seem “natural” and seems to
be a large additional work load.
What is YOUR Idea of a Quality Admissions
Department? Of quality student service?
Quality: What Is It?
 Quality of service: the concept does not
possess substance or mass.

Cannot place quality on the table to
analyze or study.
 Any definition of quality almost always
seems deficient
Quality is in the eye of the beholder.
 Quality definitions reflect the tastes and
biases of those doing the defining.

Quality
A quality admissions program may be one
in which services for students faculty and
administration increases the likelihood of
desired outcomes and is consistent with
current professional knowledge
Performance
+ Service
(-)
Wrong Things Done Right
Inefficient admission
process or system but
people do an excellent
job of entering the data
Right Things
Done Right
Wrong Things
Done Wrong
Inefficient system to
enroll students and staff
using system make many
errors entering the data
Right Things
Done Wrong
State-of-the-art equipment
used incorrectly by staff
P
R
+
O
C
E
S
S
(-)
11
Quality
Doing the right
things and
doing them
well.
Quality
 Naked assertions of high quality
admissions, records, evaluations,
counseling or any college department
or program is no longer sufficient.
Consumers, society, accrediting
agencies and campus administrators
hold us accountable to provide
objective, quantifiable standards for
quality student and service outcomes.
Admissions Assessment and
Improvement
You know, WE know, that
admission departments
help maintain the
institution and are
ESSENTIAL to operations.
Assessment will help
demonstrate that each
staff member in your
admissions/ registration
departments are
endeavoring to do their
jobs better every day.
Assessment Principles
 Assessment is NOT a one-time activity; it is
evolutionary, ongoing, incremental.
 Over time, assessment efforts should
become more comprehensive, systematic
and integrated.
 Assessment efforts must be compatible
with the institution's mission

Diane Ramirez of College of the Desert and Merrill Deming at Crafton Hills College
How Can You/We Do It All?
FERPA
Budget
Crises
Day-to-Day
Operations
Increasing
Demand
for
Services
OUTCOMES
MEASUREMENT?
What is Assessment?
Assessment is an ongoing process aimed at
understanding/improving student learning and campus
services and processes
 It serves to clarify our values
 Sets criteria and standards for learning and services
 We gather data to see how well performance
matches standards
 Assessment is deciding what YOU as THE subject
matter experts value and wish to measure and then
using the resulting information to document and
improve performance
Key Questions
The answers to these questions form the essence of what is called assessment.





What are we trying to do?
What is it that admissions should accomplish?
How is success defined?
How successful are our students?
How well have they learned?

How do we know?
 How well are we performing our functions?
 How do we know?
 How do we use that information to continually
improve?
 Does the improvements we make work and do we
maintain the gain?
Adapted From M. Bresciani, Director of Assessment SDSU and from CUPR Guidelines
Student Learning Outcomes
Definition
 The knowledge, skills,
abilities, attitudes, habits of
mind or competencies that
students have at the
completion of a
 Course
 Program
 Activity or interaction
The presence and quality
of which can be measured
and evaluated.
Categorization of Student
Learning Outcomes
 Complex Cognitive Skills
 Knowledge Acquisition
 Interpersonal





Development
Social Development
Academic Achievement
Persistence
Practical Competence
Civic Responsibility
National Assoc. of Student Personnel Administrators, Upcraft 1999
Assessment to Improve
Admissions Processes/Services
 Redesign services based upon best





practices
Use information technology to improve
access to information and to support
decision-making
Improve workforce knowledge and skills
Develop effective teams within and outside
the department(s)
Coordinate referrals
Measure performance and outcomes
Why Do We Use An
Assessment Framework?
Model provides common language and a
consistent framework
 Allows celebration of small wins
 Demonstrates accomplishments
 Is readily explained, one page
 Model clearly shows linkages with larger
institutional competencies and college
mission

San Diego City College’s Six
Column Model
 Credit to Jim Nichols – We have adapted his work
with over 380 colleges and are using an expansion of
his model.
 Columnar model--helps individuals see linkages
between levels of organization.
City College Six Column
Assessment Model
 Column 1: Statement of Institutional Purpose—for City this is our
Mission and our Institutional Priorities
 Column 2: Institutional Competencies











Communication/Interpersonal Skills
Critical thinking
Analyses/Computation
Cultural Sensitivity/Global Awareness
Information Management/ Literacy
Personal Responsibility
Civic and Environmental Responsibility
Column 3: Programmatic Student Learning Outcomes
Column 4 : Means of Assessment and the Criteria for Success
Column 5: Summarizes data collected (Results)
Column 6: Designated to demonstrate how you will use results to
improve the program
Note: Linkages between the 6 columns is essential
Mission
 Your departmental
mission is your reason for
being, the motivation
for doing this work, the
soul of the department
Column One
 To guide City faculty and staff in
formulating outcomes, column one is
comprised of the City College mission
statement and our college priorities
Admissions/Registration 2009
Outcome
COLUMN
ONE
College
Mission and
Priorities
Column
Institutional Competencies
 Communication/Interpersonal Skills
 Critical thinking
 Analyses/Computation
 Cultural Sensitivity/Global Awareness
 Information Management/ Literacy
 Personal Responsibility
 Civic and Environmental Responsibility
Admissions/Registration 2009
Outcome
College
Mission
and
Priorities
COLUMN 2:
Institutional Competencies
Communication/Interpersonal
Skills


Critical thinking

Analyses/Computation
Cultural Sensitivity/Global
Awareness

Information Management/
Literacy


Personal Responsibility
Civic and Environmental
Responsibility

Column :Development of
Departmental/Programmatic SLO’s
 Departmental staff members
convene to discuss and reach
consensus on the focus of
assessment for the coming
semester/year.
Developing Programmatic Student
Learning OR Administrative Unit
Outcomes
 Collect and review documents that





describe the program (catalog, your
program’s mission)
Examine outcome statements from
similar programs at other institutions
Review assessment criteria of WASC
accrediting body
Review criteria from your professional
organization--CACCRAO
Collect and review departmental
materials
Brainstorm with your staff and utilize the
short list/long list concept
Long and Short List Concept

Make a LOOONG list
of all the learning or
administrative
outcomes of
importance to your
department:
1.
2.
3.
…..
42.

Now choose 1-2 to study
this semester/year. Have
your department
collaboratively discuss
and reach consensus on
what is important THIS
year or semester:
1.
2.
Writing Outcomes
 Uses clear, observable and measurable verbs
 Compile
Arrange
Classify
 Analyze
Identify
Operate
 Design
Solve
Write
 Apply
Differentiate
Calculate
 Demonstrate
Formulate
Compose
 Explain
Predict
Assess
 Compare
Estimate
Critique
 The following verbs are relatively vague states of mind. They are
very difficult/ impossible to measure:

Know
Understand
Credit to Gavilan College for this content
Appreciate
Learn
Admissions/Registration 2009
Outcome
College
Mission and
Priorities
Institutional Competencies
COLUMN 3
Communication/Interpersonal
Skills
Programmatic
Learning or
Administrative
Outcome

Critical thinking

Analyses/Computation

Cultural Sensitivity/Global
Awareness

Information Management/
Literacy

Personal Responsibility

Civic and Environmental
Responsibility

After an encounter in
the Admissions office
or through the website
students will
______________________
______________________
_
Or the Admissions
Dept.
will___________________
______________________
______
Column 4—Means of Assessment
and Criteria for Success
 “Rubber hits the road” in this column
 Here you are defining the specifics of
measurement and precisely what you meant in
column 3. You are describing the context for the
outcome. What do you mean that you want the
student to be better at availing him/herself of
campus resources? Able to navigate the
catalog?
 It is in this column you must be very clear on the
specifics of measurement, and at what point you
will consider yourself successful. 78% of the
students will…..etc..
Admissions/Registration 2009
Outcome
College
Mission and
Priorities
Institutional Competencies
COLUMN 3
COLUMN 4:
Communication/Interpersonal
Skills
Programmatic
Learning or
Administrative
Outcome
Means of Assessment and
Criteria for Success

Critical thinking

Analyses/Computation

Cultural Sensitivity/Global
Awareness

Information Management/
Literacy

Personal Responsibility

Civic and Environmental
Responsibility

After an
encounter in
the Admissions
office or
through the
website
students will
_______________
_______________
_______________
Or the
Admissions
Dept. will
decrease
turnaround
time for ____
filing.
Using the 2009 Admissions
satisfaction survey 72% of
students will demonstrate
3.5 or above on the
satisfaction scale of the
admissions dept.’s ability to
assist them with ________.
Using computer data
from_____the Admissions
dept. will demo a 12.6%
decrease in turn-around
time for ______ in the 20092010 school year.
 Column 5: Log results obtained and compare to the
criteria for success that the department or program
set for itself
 Column 6: Compare how actual results stacked up
against the criteria for success. Will the department
rotate the outcome off the list as “accomplished” and
not problematic, or does it see the issue as an
opportunity for programmatic improvement?
 What are the departmental actions to improve?
 NOW you have “closed the loop” and the entire goal
of this assessment effort comes full cycle: you now
have data and actions to support program
improvement efforts.
Admissions/Registration 2009
Outcome
College
Mission
and
Priorities
Institutional Competencies
COLUMN 3
COLUMN 4:
Communication/Interpersonal
Skills
Programmatic
Learning or
Administrative
Outcome
Means of
Assessment and
Criteria for
Success
After an
encounter in the
Admissions office
or through the
website students
will
________________
________________
_____________
Using the 2009
Admissions
satisfaction
survey 72% of
students will
demonstrate 3.5
or above on the
satisfaction scale
of the admissions
dept.’s ability to
assist them with
________.

Critical thinking

Analyses/Computation

Cultural Sensitivity/Global
Awareness

Information Management/
Literacy

Personal Responsibility

Civic and Environmental
Responsibility

Or the
Admissions Dept.
will decrease
turnaround time
for ____ filing.
Using computer
data
from_____the
Admissions dept.
will demo a
12.6% decrease
in turn-around
time for ______ in
the 2009-2010
school year.
COLUMN 5:
Results
Using the 2009
Admissions
satisfaction
survey 64% of
students
demonstrated 3.5
or above on the
satisfaction scale
of the admissions
dept.’s ability to
assist them with
________.
Using computer
data
from_____the
Admissions dept.
demo’ed a 4%
decrease in turnaround time for
______ in the
2009-2010 school
year.
COLUMN 6: Use
of Results for
Program
Improvement
With these results,
we have
decided to:
Do nothing
because our
findings are so
good. We will
move on to
something else.
OR
We will do
________ to
improve our
findings and will
re-measure
again in _____
2009/2010.
Column 6: Closing the Loop
Key Questions
 How well are we
achieving the
admissions/registrar
program or department
goals?
 How do we know?
 How can we use
data/information to
continually improve the
processes and systems
of the department?
Student Learning Outcomes And Administrative
Unit Outcomes: Clarification: FAQ’s
What is a student learning outcome (SLO)?
Knowledge, skills, abilities, attitudes, habits of mind that students have at
the completion of a series of activities or interactions--the presence and
quality of which can be measured and evaluated
What is an AUO (Administrative Unit Outcome)?
An AUO describes what an “Administrative Unit” such as your Admissions
Depts. will accomplish or do in the provision of their services. The focus of
AUOs is services. How you know you are accomplishing the purpose of
Admissions, Records, Registration? THE FOCUS IS ON IMPROVEMENT IN
SERVICES PROVIDED TO SUPPORT STUDENT LEARNING.
Why is it important to include student support services in this work?
Key to understanding the importance of this initiative is the premise that
the entire campus supports and is crucial to student learning. As such it
requires a commitment of the entire campus as part of a comprehensive
Institutional Effectiveness program
Why should I engage in outcome work if I’m not in an instructional area of the
campus?
Student learning experiences most definitely do not just occur in the
classroom. Some of the most important learning on campus occurs
outside of the classroom. It is important that the campus participate to
demonstrate learning outside the classroom and the efficiency of campus
operations.
Similarities between SLOs and AUOs
 Both focus on improvement (in either
student learning or administrative
functioning)
 Both contribute to and are part of the
quality picture of the entire campus
 Both use assessment data to effect
improvement
Examples Of Programmatic
Outcomes— Admissions
Admissions will shorten the response time for _____from 3 days to 2.5 days .
Over the counter requests for __________will be delivered in ____minutes/hours
versus ____minues/hours.
 94% of students will articulate a high level of satisfaction with the overall
registration process as evidenced by an increase from 2.7 to 3.5 on a 5 point
satisfaction scale.









Or 85% of students will demonstrate a high( remember to define what you mean by
“high”) level of satisfaction on exit interviews from the admissions dept.
Admissions office complaints or dissatisfaction reports will decrease from ___
per semester to <2 per semester.
Admissions professionals will contribute to the overall campus commitment to
a diverse student body by increasing_____________.
The Admissions department will institute the following interventions this
semester to increase the % students having a valid ID card prior to “x” date of
the semester from ___% to ___%:
Admissions will demonstrate an increase in students successfully( how is
successfully defined?) navigating the college catalogue from ___% to ___%
by______________________________.
To improve the learning outcome related to services referral and campus
networking---As reported on an exit questionnaire from the Admissions
department, students will accurately report 3 of 5 departments necessary to
meet their needs and educational goals.
Admissions will provide accurate class enrollment data_____________.
Key Foundational Assessment
Factors
at City College
 Not all outcomes are student learning outcomes, some may be
administrative outcomes which are essential to, and form the foundation
upon which, student learning and success are built
 This whole process is directed towards incremental program improvement,
not individual evaluation
 The “program” is as the department/area/service defines it.
 Key themes emerged at City on learning outcomes and Student Services:


Students learn about other campus resources from those of us in Student Services
Students learn from Student Services professionals how to better advocate for themselves



Self-advocacy
Self-assertiveness
Networking
 To have buy-in, sustainability, and assessment of value-- it must be an
inclusive staff/faculty driven process: a grassroots effort
 Values driven
 You and your department are the subject matter experts —what do
YOU feel is important to study in your department?
 Part of daily/weekly/monthly/semester work--not a set-aside project
Student Services
 The campus is our classroom.
 All of us are teachers and each of us is
essential to the college mission and the work
and learning that occurs daily with students.
 Assessment is good---it demonstrates for us
the vital role that each of us play in the
educational process.
How To Start
 What do you and your dept. value or feel is
important to do well in your area?
 Seek consensus in your area on choosing this issue
to study.


Pick one for now.
Keep it simple
 Write it down
 Decide on a measure
 Observe students?
 Survey?
 Collect data
 Act on the results.
 Write it down.
Progressive improvements beat
postponed perfection every time
Direct and Indirect Measures
 Satisfaction surveys
 Provide feedback on student opinions
 Indirect measure of assessment
Measure what the student feels or thinks about the
program
 Move away from satisfaction surveys
 Optimal to use satisfaction measures in conjunction
with other measures
 The assessment effort seeks to find what the students
know, have learned or can demonstrate
 Direct measures of learning

Tips in Creating a Survey
 Limit number of questions
 Include explicit directions
 Check reading level
 Eliminate having two questions built
into one
 Pilot the survey
 Consider alternative methods of
distribution/collection
 Consider whether anonymity is an
issue
Lessons from the Field

A thousand mile march really does begin with your first
step.

Change is rarely convenient. The only who likes it is a
wet baby.

Who are the assessment champions? Persons willing to
be accountable to keep the process going forward?
Set meetings? Keep notes? HOPE IS NOT A PLAN.

Focus, Focus, Focus. Start with just one question, one
form, one anything!

KNOW how you will know that what you have done is
an improvement. Design for incremental
improvement, not publication or perfection.

Don’t get hung up on names: outcomes, objectives,
competencies, goals, SLO’s, AUO’s, program
definition. You know what is important to your dept.


Sometimes you just have to jump in and DO IT!
You are doing assessment much more than you
think…you just are not taking credit for it.
Contact Information
Dotti Cordell
San Diego City College
1313 Park Blvd. Rm. A116
San Diego, CA 92101
(619) 388-3903
Email: [email protected]