Transcript 360 Degree Academic Performance Assessment Model

Southeastern Association
for Community College Research
2008 Conference
Using Technology to Efficiently and
Effectively Gather Information from
Recent Alumni and Employers
August 2008
Alumni and Employer Surveys
SACCR 2008 Conference
August 2008
Presenters
 Dr. James Coraggio, Director of
Institutional Research and Effectiveness
 Maggie Tymms, Assessment Coordinator
for Academic Programs
August 6, 2008
Southeastern Association for Community College Research
2
SPC Background
SACCR 2008 Conference
August 2008
 SPC, established in 1927, is the oldest
2-year college in Florida
 First Community College in Florida to offer
4 year degrees (2002)
 9 Campuses throughout the county
 2007-08 FTE: 16,086 (LD), 1,338 (UD)
 Opening Spring 2008 headcount: 25,754
 Annual 2007-08 enrollment: 62,200
August 6, 2008
Southeastern Association for Community College Research
3
Institutional Effectiveness
SACCR 2008 Conference
August 2008
“Closing the Loop”
Mission
Evaluations/ Strategic & Unit
Assessments
Planning
Budget Planning Operating Results
“Institutional Effectiveness is the integrated,
systematic, explicit, and documented
processes of measuring performance against
the SPC mission for purposes of continuous
improvement of academic programs,
administrative services, and educational
services offered by the college.”
August 6, 2008
Southeastern Association for Community College Research
4
Performance Improvement
SACCR 2008 Conference
August 2008
 From Compliance to Performance Improvement
Community
College Leadership
Program Directors
Faculty
Students
Assessment
Data
Reports
SPC
Data
Information
Knowledge
August 6, 2008
Improved Outcomes
Improvement
Southeastern Association for Community College Research
5
Agenda
SACCR 2008 Conference
August 2008
 Student Assessment Points
 Purpose of Follow-up Surveys
 Alumni/Employer Survey Process
 Dissemination of Survey Results and
Reports
August 6, 2008
Southeastern Association for Community College Research
6
Student Assessment Points
SACCR 2008 Conference
August 2008
Students surveyed multiple times
August 6, 2008
Southeastern Association for Community College Research
7
Purpose of Follow-up Surveys
SACCR 2008 Conference
August 2008
Recent Alumni Survey was designed to:
 Gain insight into alumni’s perceptions of
the quality of general education and career
preparation
 Determine the current activities of former
students such as employment information
and educational status
August 6, 2008
Southeastern Association for Community College Research
8
Purpose of Follow-up Surveys
SACCR 2008 Conference
August 2008
Employer Survey was designed to:
 Gain insight into an employer’s perceptions
of hired SPC graduates in the areas of key
general education competencies and
foundational skills
 Provide the employee’s major job
responsibilities and wage information, as well
as employer’s willingness to hire another
SPC graduate
 Identify employer’s willingness to support
various college activities
August 6, 2008
Southeastern Association for Community College Research
9
Purpose of Follow-up Surveys
SACCR 2008 Conference
August 2008
State Requirements
 Survey data are also collected for state
required annual Placement & Follow-up
Report (FETPIP: Florida Education
Training Placement Information Program)
and supplemental information to meet
workforce criteria
August 6, 2008
Southeastern Association for Community College Research
10
Alumni/Employer Survey Process
SACCR 2008 Conference
August 2008
Previous model
 Labor intensive
 Manually tabulated paper surveys
 Telephone interviews conducted
using overtime hours
August 6, 2008
Southeastern Association for Community College Research
11
Alumni/Employer Survey Process
SACCR 2008 Conference
August 2008
Issues with previous model
 Costly survey process
 Declining college budget
 Increasing demand for accountability
 Outdated contact information
 Students over-surveyed
 Declining response rates
 Conducted in isolation
August 6, 2008
Southeastern Association for Community College Research
12
Alumni/Employer Survey Process
SACCR 2008 Conference
August 2008
Goals of new survey process
A. Improve data collection process
B. Enhance quality of survey forms
C. Include program specific
information
D. Increase response rates
August 6, 2008
Southeastern Association for Community College Research
13
A. Improve Data Collection Process
SACCR 2008 Conference
August 2008
Data Collection Process
 Eliminated telephone calls
 Automated the process (email merge)
 Employ online survey submission
(Zoomerang)
 Obtain accurate email addresses
 Use multiple points of contact
August 6, 2008
Southeastern Association for Community College Research
14
A. Improve Data Collection Process
SACCR 2008 Conference
August 2008
Survey Schedule
 Recent Alumni Survey
 Graduates surveyed six months after
the end of session (Three times per
year)
 Employer Survey
 Employer of graduates surveyed at the
end of academic year (Once per year)
August 6, 2008
Southeastern Association for Community College Research
15
A. Improve Data Collection Process
SACCR 2008 Conference
First email
August 2008
Second email
2 weeks
Third email
2 weeks
2 weeks
Multiple Points of Contact
2 weeks
First mailing
August 6, 2008
Second mailing
Southeastern Association for Community College Research
16
B. Enhance Quality Of Survey Forms
SACCR 2008 Conference
August 2008
Improved Survey Forms
 Better “look and feel” of survey format
 Developed scannable paper forms
 Use online branching models to reduce
survey length
 Design items to gather only pertinent
data
August 6, 2008
Southeastern Association for Community College Research
17
Online Survey: Email and Form
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
18
Paper Survey: Letter and Form
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
19
C. Include Program Specific Information
SACCR 2008 Conference
August 2008
Program specific information
 Meet with PDs
 Discuss requirements for program
accreditation and timelines
 Include their necessary program specific
items
August 6, 2008
Southeastern Association for Community College Research
20
D. Increase Response Rates
SACCR 2008 Conference
August 2008
Create awareness and convey importance




Involve program faculty in the process
Request faculty assistance to inform students
Include a flyer with graduation materials
Coordinate with programs that hand-carry
surveys to local employers
August 6, 2008
Southeastern Association for Community College Research
21
Response Rates
SACCR 2008 Conference
August 2008
 Summer 07
August 6, 2008
Southeastern Association for Community College Research
22
Response Rates
SACCR 2008 Conference
August 2008
 Fall 07
August 6, 2008
Southeastern Association for Community College Research
23
Response Rates
SACCR 2008 Conference
August 2008
12.00%
11.2%
10.1%
10.00%
9.1% 9.0%
8.3%
7.6%
8.00%
6.4%
6.3%
6.00%
5.2%
5.6%
4.2%
5.3%
4.4%
4.00%
2.00%
0.00%
1st E-mail
2nd E-mail
Summer & Fall 06
August 6, 2008
3rd E-mail
Spring 07
1st Mailing
Summer 07
2nd Mailing
Fall 07
Southeastern Association for Community College Research
24
Response Rates
SACCR 2008 Conference
August 2008
Overall Response Rate
23.4%
25.0%
21.2%
19.0%
20.0%
13.7%
15.0%
10.0%
5.0%
0.0%
Summer & Fall 06
August 6, 2008
Spring 07
Summer 07
Fall 07
Southeastern Association for Community College Research
25
Response Rates - Employer
SACCR 2008 Conference
Recent Alumni
Surveys Sent
August 2008
Is a 44% Employer
Response Rate Good?
COE
Responses
Other
Responses
August 6, 2008
Employer
Surveys Sent
With Permission
Responses
Southeastern Association for Community College Research
26
Disseminating Information
SACCR 2008 Conference
August 2008
 College wide access to reports
 http://www.spcollege.edu/Central/Surveys/SurveyList.htm
 Program Specific Reports
 Published on the Educational Outcomes
website
 http://it.spcollege.edu/edoutcomes
August 6, 2008
Southeastern Association for Community College Research
27
Disseminating Information
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
28
Disseminating Information
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
29
Disseminating Information
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
30
Program Specific Reports
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
31
SPC IE Website
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
32
Lessons Learned
SACCR 2008 Conference
August 2008
 Do not expect that students remember their
ID#
 Administer the survey soon after the end of
the session (six months)
 Use multiple points of contact (two weeks)
 Find the most recent contact information
 Make students aware of the importance of
the survey process (survey announcements)
August 6, 2008
Southeastern Association for Community College Research
33
Lessons Learned
SACCR 2008 Conference
August 2008
 Partner with programs to acquire up-todate information
 Provide updates to college data systems
(PeopleSoft, Alumni Foundation)
 Monitor results and make changes as
necessary
 More accurate data (quicker turn around of
results) with automated processes
August 6, 2008
Southeastern Association for Community College Research
34
Future Direction
SACCR 2008 Conference
August 2008
 Ultimate goal is to provide Provosts
(Deans) and Program Directors ‘timely’,
‘relevant’ ‘accurate’, and ‘interpretable’
information regarding their programs and
their performance outcomes
 Multi-measure ‘Dashboard’ style reports
(Business Intelligence/Performance
Measurement)
August 6, 2008
Southeastern Association for Community College Research
35
Questions/Next Steps
SACCR 2008 Conference
August 6, 2008
August 2008
Southeastern Association for Community College Research
36
Southeastern Association
for Community College Research
2008 Conference
Using Technology to Efficiently and
Effectively Gather Information from
Recent Alumni and Employers
August 2008