Transcript How can IR help with assessment and retention? A presentation for the Oklahoma Association for Institutional Research and Planning by Scott E.

How can IR help with
assessment and retention?
A presentation for the Oklahoma Association for
Institutional Research and Planning
by
Scott E. Bodfish
Vice President
May 5, 2009
All material in this presentation, including text and images, is the property of Noel-Levitz, Inc. Permission is required to reproduce information.
Today’s Agenda
• Outline of a process for working with
campus retention and assessment
initiatives
• Case studies to illustrate that process
• Presentation of some strategies for
effective research
• NOTE: Please mute your phone
– *6 to mute; #6 to unmute
So, your provost ordered the raw data
from your satisfaction/priorities survey
Here is what she got!
Now she wants you to tell her what it all
means
Although it will
be pleasant you
can waste a lot
of time analyzing
the data without
some kind of
plan
The Research Design Process
Define the problem
Develop a research objective
Collect data
Turn data into information
Get information to decision-makers
The Diagnostic Process
There is a problem (symptoms)
The symptoms have an underlying cause
The nature of the cause allows for intervention
Implementing the intervention
Example
The number of students failing introductory statistics
courses has increased.
Many of them have lower scores on basic math
skills (specifically algebra) than previous cohorts of
students.
We can train advisors to counsel students into
remedial math based on assessment test score.
Administer assessment tests; train advisors; monitor
statistics course pass rates.
Developing targeted counseling
services at a four-year public university
Counseling and advising of new students doesn’t
seem to be working well
All students are going through the same process; no
customization
We need a means for assessing student needs that
allow for customization
Identify an instrument; administer and prioritize
services accordingly
The situation
• Program serves at-risk students
(economically disadvantaged; inadequate
academic preparation)
• Sought an instrument to assess student
needs for intervention and support beyond
academic remediation
• CSI not only identifies needs but assesses
receptivity to help
Findings from the research
• 20% of entering freshmen had plans to
work 21-30 hours per week
– Include two time management seminars in our
summer programming
– Counselors know to specifically speak with
students about work and the time required to
be successful in their studies
Findings from the research
• 46% of the initial respondents indicated
that the highest degree they sought would
be a graduate degree
– A workshop on graduate studies was
introduced for students in their second year
rather than at the end of the third year
Findings from the research
• Women more receptive to academic
assistance and personal counseling while
men more receptive to career counseling
– Counselors structure the topics differently for
men than for women; however
– Individuals vary within each gender group so
the primary focus is on each individual’s
specific receptivity
Outcomes
• Now provide in-classroom supplemental
instruction for 2 levels of math for students
identified with this need
• Hired two academic coaches to assist
students who scored high on motivation
and receptivity scales, but low on
confidence and study skills
• An increase in the “show rate” to 87% for
all booked coaching sessions
Improving facilities and services at a
community college
Rapid enrollment growth and general complaints
about student service / satisfaction
No way of prioritizing student service initiatives
Identify and prioritize specific concerns
Ongoing measurement of levels of satisfaction
The situation
• Selected SSI over “home grown”
instrument in order to use a standardized
instrument used for self-study
• Have administered SSI every other year
since 1996
• Administration procedures have remained
consistent over time
Use of Data to Make Changes
• Created new library (information
commons) in late 1990’s
– Satisfaction w/ library services increased from
5.22 to 5.64
• Created new student life center
– Satisfaction (it is enjoyable to be a student on
this campus) increased from 4.72 to 5.41
Use of Data to Make Changes
• Consolidated counseling services
– Slight increase in satisfaction with advising
• Incorporated student development
services (admissions, financial aid,
registration, career services, disability
services) along w/ counseling
– Much greater increase in satisfaction
• Changed course scheduling process
– Second largest increase in satisfaction
Use of Data to Make Changes
• Added parking in 1998
– Satisfaction increases from 3.35 to 3.81
• Enrollment increases
– Satisfaction decreases to 3.54
• Added more parking in 2002
– Satisfaction increases to 4.31
• Enrollment has increased again
– Satisfaction has declined to 3.75
Understanding attrition at a four-year
private university
An increase in attrition rates
Engagement and academic performance (not
finances) related to withdrawal
Change marketing / recruiting messages to create
more accurate expectations
Ongoing measurement of attrition
The plan before research
• Private, four-year institution (high cost)
• Anecdotal evidence that students feel they
aren’t receiving enough financial aid
• We should increase institutional financial
aid packages for continuing students
Financial considerations the most
common reason for leaving college
Financial reasons
40%
Other
35%
Family responsibilities
30%
Class not available / scheduling
inconvenient
Dissatisfaction with program / school /
campus / faculty
Completion of degree / certificate
25%
20%
15%
Academic problems
10%
Finished taking desired classes
5%
Personal health reasons
0%
Reasons for discontinuing
postsecondary education
Traumatic experience
Military service
SOURCE: ELS:2002 “A First Look at the Initial Postsecondary Experiences of the High School
Sophomore Class of 2002 (National Center for Education Statistics)
The research project
• Structured interviews with 20 students (10
who persisted and 10 who withdrew)
• Analysis of institutional historical data files
(two cohorts)
– Data included EFC, financial aid awards,
admissions data (HS GPA, SAT scores),
academic performance data (credits
registered, credits completed, GPA earned)
Findings from the research
• Some students, especially freshmen, seem to
have a low initial commitment to the university.
• Academic performance in the first semester sets
the tone for most students.
• Financial need is unrelated to whether or not a
student will withdraw
• Basic measures such as high school GPA and
class rank provide a better indication of which
students will have difficulty academically
$0
$8
9
$3
52
$6
46
$9
$1 74
,2
$1 21
,4
$1 79
,5
$1 84
,8
$2 38
,1
$2 11
,4
$2 70
,8
$3 86
,2
$3 30
,7
$3 29
,9
$4 28
,2
$4 73
,7
$5 86
,5
$5 78
,9
$6 74
,4
$6 14
,7
$7 79
,3
$7 60
,8
$8 99
,4
$9 00
,
$1 361
0,
$1 085
1,
$1 269
1,
$1 752
2,
$1 381
3,
$1 062
4,
$1 269
6,
$1 532
7,
$1 502
8,
$2 598
2,
03
4
Cumulative Percent Attrition
Fall 03 Freshmen not Returning Fall 04
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
Expected Family Contribution
Difference between Predicted and Actual GPA
1.59
1.27
1.13
1.05
0.94
0.83
0.76
0.70
0.66
0.61
0.57
0.52
0.47
0.39
0.32
0.25
0.20
0.13
0.08
0.01
-0.03
-0.09
-0.15
-0.23
-0.33
-0.40
-0.45
-0.52
-0.63
-0.69
-0.83
-1.03
-1.25
-1.35
-1.56
-1.93
-3.21
Cumulative Percent Attrition
Fall 03 Freshman Not Returning Fall 04
100%
90%
80%
70%
60%
50%
40%
30%
20%
10%
0%
So what about
that survey data
the provost
requested?
Why focus on student satisfaction?
Graduation Rate by
Satisfaction Quartile
60
50
40
30
20
10
0
1
2
3
4
5
Satisfaction Quartile
•Schools with higher
overall satisfaction
enjoy:
– Higher graduation
rates
– Lower loan default
rates
– Higher alumni giving
rates
Identifying the
“drivers” of
satisfaction
Tuition paid is a
worthwhile
investment
1
2
3
4
5
6
7
All in all, if you had to do it over
again, would you enroll here?
1
2
3
4
5
6
7
2% 0% 0% 1% 0% 1% 0%
1% 1% 1% 0% 0% 1% 0%
0% 3% 1% 2% 1% 3% 1%
0% 1% 1% 1% 1% 3% 3%
0% 1% 1% 4% 3% 8% 6%
0% 1% 0% 2% 1% 9% 9%
0% 1% 0% 0% 1% 4% 14%
Parking lots are
well-lighted and
secure
1
2
3
4
5
6
7
All in all, if you had to do it over
again, would you enroll here?
1
2
3
4
5
6
7
0% 0% 0% 0% 1% 1% 1%
0% 2% 0% 0% 0% 1% 1%
1% 1% 1% 2% 1% 4% 4%
0% 1% 1% 1% 1% 4% 2%
1% 1% 1% 3% 3% 9% 9%
1% 1% 1% 3% 2% 6% 10%
0% 1% 0% 1% 1% 3% 6%
Admissions counselors
respond to prospective
students' unique needs and
interests
1
2
3
4
5
6
7
All in all, if you had to do it over
again, would you enroll here?
1
2
3
4
5
6
7
0% 0% 0% 0% 0% 0% 0%
0% 0% 0% 0% 0% 0% 0%
0% 0% 0% 1% 1% 0% 0%
1% 2% 1% 2% 3% 4% 4%
1% 1% 1% 3% 1% 6% 4%
0% 1% 1% 2% 3% 10% 12%
0% 1% 1% 2% 2% 8% 16%
What other variables influence a
student’s college experience?
What are some
strategies for
working effectively
with campus
retention or
assessment
initiatives?
Be clear on
institutional
priorities
There are regular tasks that aren’t
strategic but still high priority
Cultivate a
relationship with
the retention or
assessment
team
Brainstorm with them to get their ideas
Make joint requests for resources
Share the
successes
Relate specific results to insights from
the data
What kind of
IR / Planning
shop do you have?
We just fill out
institutional surveys
Important to have consistent
institutional data reported
There are more surveys than there is
time available
You may be able to “mine” the surveys
for data you need
We just manage
surveys or extract
data from internal
databases
Is it time to re-visit the survey calendar in
light of retention or assessment initiatives?
Are there regular reports that provide
information for retention or assessment?
The “internal
consultant”
Tends to be entrepreneurial
More often reports to the president than
to a vice president
May have a specialized focus or
interest
There are always more data to analyze, more
studies to design and more reports to write
Having a process that begins with defining
the problem will help clear the way