Transcript Slide 1

Southeastern Association
for Community College Research
2008 Conference
Beyond Compliance…Using a
Model of Comprehensive
Academic Program Review to
Improve Education
Comprehensive Academic
Program Review
Department of Institutional
research and Effectiveness
St. Petersburg College
P.O. Box 13489,
St. Petersburg, FL 33733
(727) 341-3059
FAX (727) 341-5411
Presenters
•James Coraggio, Director of Institutional
Research and Effectiveness
•Maggie Tymms, Assessment Coordinator for
Academic Programs
August 2008
Southeastern Association for Community College Research
2
SPC Background
 SPC, established in 1927, is the oldest
2-year college in Florida
 First Community College in Florida to
offer 4 year degrees (2002)
 9 Campuses throughout the county
 2007-08 FTE: 16,086 (LD), 1,338 (UD)
 Opening Spring 2008 headcount: 25,754
 Annual 2007-08 enrollment: 62,200
August 2008
Southeastern Association for Community College Research
3
Compliance Issues with IE
 SACS’s Peer Review Research
Project revealed that in the area
of Institutional Effectiveness (IE)
 62% of institutions in off-site
reviews were deemed noncompliant and
 27% of institutions in on-site visits
were deemed non-compliant
 IE was the second most identified
area behind faculty qualifications
August 2008
Southeastern Association for Community College Research
4
SPC Experience with IE
 As part of the recent SACS
reaccreditation process, SPC spent a
considerable amount of time and
effort documenting and detailing their
IE processes
 The result…SPC had “zero” compliance
issues in the area of IE during the offsite reviews
August 2008
Southeastern Association for Community College Research
5
SACS Requirements for IE
 Core Requirement 2.5: Integrated Planning
& Evaluation Processes
 “…ongoing, integrated, and institution-wide
research-based planning and evaluation
processes…”
 Comprehensive Standard 3.3.1: Institutional
Effectiveness
 “…identifies expected outcomes…; assesses
whether it achieves these outcomes; and
provides evidence of improvement…
 Comprehensive Standard 3.5.1: General
Education Core
 “… identifies college-level competencies within
the general education core and provides evidence
that graduates have attained those
competencies.”
August 2008
Southeastern Association for Community College Research
6
SACS Requirements for IE
 Additional IE related requirements
include:
 Comprehensive Standard 3.1.1:
Institutional Mission
 Federal Requirement 4.1: Evaluating
Success
 Federal Requirement 4.2: Curriculum
August 2008
Southeastern Association for Community College Research
7
IE Evidence
 SACS Suggested Documentation for CR 2.5
 Evidence of linkage of IE to institutional mission
 Institutional plans and budgets that demonstrate
linkage of assessment findings to planning at all
levels
 Minutes of appropriate (IE related) unit,
committee, task force meetings…
 Documentation that relates to IE, such as budget
preparation instructions, minutes of budget
presentation meetings, annual reports, annual
assessment updates, IE reports
 Samples of specific actions taken to improve the
IE process and/or results from that process
August 2008
Southeastern Association for Community College Research
8
Performance Improvement
 From Compliance to Performance
Improvement
C o m m u n ity
C o lle g e L e a d e rs h ip
P ro g ra m D ire c to rs
F a c u lty
S tu d e n ts
A ssessm ent
D a ta
R e p o rts
SPC
D a ta
In fo rm a tio n
K n o w le d g e
August 2008
Southeastern Association for Community College Research
Im p ro v e d O u tc o m e s
Im p ro v e m e n t
9
IE at SPC
“Closing the Loop”
M issio n
E va lua tio ns/
A sse ssm e n ts
S tra te g ic & U n it
P la nn ing
B u dg e t P lan n ing
O p e ra tin g R e su lts
“Institutional Effectiveness is the
integrated, systematic, explicit, and
documented processes of measuring
performance against the SPC mission for
purposes of continuous improvement of
academic programs, administrative
services, and educational services
offered by the college.”
August 2008
Southeastern Association for Community College Research
10
Evaluation & Assessment
 Evaluation and Assessment are
about ‘Choices, Direction,
Focus…Quality Improvement’
• Instructional Effectiveness
Processes at SPC were recently
evaluated as part of the College’s
on-going continuous performance
improvement
August 2008
Southeastern Association for Community College Research
11
Educational Assessment
• One IE process that experienced
extensive revisions was educational
assessment
• Individual assessments in the areas of
academic programs, Gen Ed, and
program review were evaluated and
enhancements were made to the
quality and integration of assessments
as well as the dissemination of
assessments results
August 2008
Southeastern Association for Community College Research
12
Changes & Improvements
 Evaluated individual assessment instruments
and current assessment processes
 Found new ways to integrate assessment
with the educational process
 Focus on fewer, yet meaningful
improvements (action items)
 Use multiple assessment methods (direct
& indirect)
 Involve faculty in process
 Integrate timing of assessments
(3-year cycle)
August 2008
Southeastern Association for Community College Research
13
Three-year Assessment Cycle
APARs
Academic Program
Viability Reviews
(APVRs) conducted yearly
August 2008
CAPRs
Southeastern Association for Community College Research
14
Changes and Improvements
 Now looking at the ‘Big Picture’
 Assessment focus is on CHANGE
through Quality Improvement
 Faculty-driven assessment process; IRE
serves in consultant capacity
 Focus on critical major learning
outcomes (MLOs) with fewer, more
focused action plan items
 Improve access and awareness of
assessment information
August 2008
Southeastern Association for Community College Research
15
SPC Assessments
SPC Mission – Assessments - Closing the Loop
SPC Mission Components
Accessible Education






1.
BS, BAS, AA, AS Degrees
Technical Certificates
Applied Tech. Diplomas
Distance Education
Developmental Education
Continuing Education
Partnerships / Services




3.
4.
5.
Recurring Cycle
1.
2.
3.
4.
5.
1.
2.
2.
2.
3.
Environmental Scans
Trends Analysis
WorkNet Surveys & Exit Interviews
Southeastern Association for Community College Research
Annual
Every 3-years /
program
Annual
Every 3-years /
program
Annual
Annual
Annual
3. Annual
4. Annual
5. Annual
6. Annual
7. Annual
8. Annual
9. Annual
10. Annual
11. Annual
1.
1.
AS Degrees & PSAV Programs
WorkNet One-Stop Ops.
August 2008
2.
Education Oversight Group (EOG) Reports
Academic Program Assessment Reports (APARs)
Academic Program Viability Reviews (APVRs)
Comprehensive Academic Program Reviews (CAPRs)
End-of-year SD-IOs Assessment Report
Entering Students Survey
Enrolled Students Survey
3. Graduating Students Survey
4. Employer Survey
5. Recent Alumni Survey
6. Budget. Planning, IR Survey
7. Technology Resources Allocation Survey
8. Other Department-specific Surveys
9. Administrative Services Oversight (ASOG) Reports
10. Student Services Oversight (SSO) Report
11. Education Support Services (ESS) Reports (Libraries,
Learning Support Centers)
Student Services
Community Services
Other Institutions
Economic & Workforce
Development

Evaluations / Assessments
1.
2.
3.
On-going
On-going
Annual
16
Program Review Process
• New enhanced process is a
summative evaluation that
includes two reports with
multiple measures such as:
– program-specific performance
measures
– profitability measures
– economic data
August 2008
Southeastern Association for Community College Research
17
Academic Program Viability Review
 Published yearly
 Included measures:
 Program Graduates
 Course Enrollment
 Student Semester Hour
(SSH) productivity
 Relative Profitability Index
(RPI-T)
 Economic Trend Data
August 2008
Southeastern Association for Community College Research
18
CAPR Objectives
• Comprehensive Academic Program Review
(CAPR) developed to meet three objectives
within the academic assessment process:
– To provide a comprehensive report that
summarizes all elements of the program’s viability
and productivity from a 360-degree perspective
– To provide comprehensive and relevant programspecific information to key College stakeholders,
such as the President’s Cabinet members, in order
to make critical decisions regarding the continued
sustainability of a program
– To provide program leadership a vehicle to
support and document actionable change for the
purposes of performance improvement
August 2008
Southeastern Association for Community College Research
19
Previous Program Review Model
• Traditionally, program reviews at SPC
consisted primarily of a community
focus group and a few occupational
growth measures
• This information was presented to the
President’s Cabinet for evaluation
• CAPR was designed to be more
representative of a program’s quality
and, as such, contains measures
involving a number of stakeholder
perspectives
August 2008
Southeastern Association for Community College Research
20
Elements of the CAPR
• Specific CAPR measures include the
– program description with recent program
accreditation information
– program performance measures including
enrollment, productivity, grade distributions, and
fulltime/adjunct faculty ratios
– program profitability measures
– academic outcomes from recent end-of-program
assessments
– stakeholder perceptions including student surveys
of instruction results, advisory committee minutes,
and employer and recent alumni survey results
– occupation trends and information
– state graduate outcomes information
– the program director’s perspective of program
issues, trends, and recent success
August 2008
Southeastern Association for Community College Research
21
Measures CAPR
• In order to ensure that the
information presented in the
CAPR is used appropriately, each
measure includes:
– a standardized definition of the
performance measure
– a reference to the source
information used in the calculation
August 2008
Southeastern Association for Community College Research
22
Program Performance Example
Program Graduates
Program Graduates
20
17
18
14
16
Graduates
14
11
12
9
10
8
14
6
6
4
2
0
2001-02
2002-03
2003-04
2004-05
2005-06
2006-07
Figure 4: Program Graduates
Source: SPC Factbook, Table 31
August 2008
Southeastern Association for Community College Research
23
Program Profitability Example
Relative Profitability Index (RPI-T) is calculated by
dividing a program’s income by the sum of its
personnel costs and current expenses.
RPI-T
Fiscal Summary
1.00
0.90
0.80
0.70
0.60
0.50
0.40
0.30
0.20
0.10
0.00
0.61
0.43
0.31
2002-03
2003-04
0.40
0.38
2004-05
2005-06
2006-07
Figure 8: Fiscal Summary
Source: PeopleSoft Financial Production database, report ID: ORGBUDSI
August 2008
Southeastern Association for Community College Research
24
Academic Outcomes Example
Paralegal program was evaluated through an
Academic Program Assessment Report (APAR) in 20042005. Each of the program’s eight major learning
outcomes (MLOs) was evaluated.
Table 3
Summary of 2004-2005 Assessment Findings
Summary of
Assessment Findings Portfolio Evaluations
2004-2005 Session I
Evaluation Area
Number of
Students
Acceptable to
Sufficient
Florida Courts &
Documentation
Attorneys
(Mean)
(Mean)
Timely
(Mean)
9
MLO 1
9
3.83
3.85
3.89
MLO 2
9
3.59
3.78
3.78
MLO 3
9
3.70
3.78
3.81
MLO 4
9
3.78
3.81
3.81
MLO 5
9
3.70
3.81
3.81
MLO 6
9
3.52
3.81
3.70
MLO 7
9
3.59
3.81
3.81
MLO 8
9
3.81
3.85
3.85
August 2008
Southeastern Association for Community College Research
25
Stakeholder Perceptions Example
6.8
6.8
6.8
Spring 2007
Fall 2007
6.8
Fall 2006
6.9
Presentation
Spring 2006
6.6
Fall 2007
Fall 2007
Spring 2007
Organization
6.9
6.7
Spring 2007
6.8
Fall 2006
6.9
Spring 2006
6.8
6.7
Fall 2006
Spring 2007
F/S Interaction
6.8
Spring 2006
6.8
Fall 2007
6.9
6.7
Fall 2006
7.0
6.8
6.6
6.4
6.2
6.0
5.8
5.6
5.4
5.2
5.0
Spring 2006
SSI Satisfaction Level (Scale 1-7)
Purpose of the SSI survey is to acquire information on
student perception of the quality of courses, faculty,
and instruction, and to provide feedback information
for improvement. SSI: Lecture Courses
Evaluation
Content Area
Figure 9: SSI Lecture Courses
Source: PeopleSoft Query S_SSI_CHRT_QRY_CAMPUS
August 2008
Southeastern Association for Community College Research
26
Occupation Profile Example
The distribution of 2005 wage information is divided
by percentiles for hourly and yearly wages.
Table 5
Wage Information for Paralegals
Location
United States
Florida
Tampa-St.
PetersburgClearwater, FL
MSA
2005
Pay
Period
10%
25%
Median
75%
90%
Hourly
$12.63
$15.61
$19.79
$25.22
$31.41
Yearly
$26,300
$32,500
$41,200
$52,500
$65,300
Hourly
$12.36
$15.33
$19.59
$23.95
$27.42
Yearly
$25,700
$31,900
$40,700
$49,800
$57,000
Hourly
$14.25
$16.15
$19.09
$21.91
$26.25
Yearly
$29,600
$33,600
$39,700
$45,600
$54,600
Source: Bureau of Labor Statistics, Occupational Employment
Statistics Survey; Florida Agency for Workforce Innovation
August 2008
Southeastern Association for Community College Research
27
Use of Results
• To encourage the use of results, the
program director and provost are
required to provide an action plan for
improving the performance of the
program
• Follow-up report on these results is
required the following year
• CAPR process also includes a review of
the CAPR documentation by the
technical advisory committee and the
President’s Cabinet
August 2008
Southeastern Association for Community College Research
28
Action Plan Example
Action Item
Completion
Date
Responsible
Party
1
Rework website for two-year program

Include pdf worksheet for students.

Add information on ABA approval

Video clips of instructors discussing their
classes and how to be successful in that class

Add instructions on working with FACTS.org
August 2007
Program
Coordinator
2
Construct a survey to be administered to all PLA
1003 students to determine their goals in the
program/class
May 2007
Program
Coordinator
3
Develop one semester upper division classes which
will allow AA students to gain basic paralegal
information in Real Estate, Probate,
Litigation, and Family Law
June 2007
Program
Coordinator
Special Resources Needed:
Specialized data regarding library and textbook usage
Area(s) of Concern/Improvement:
The advisory committee and the instructors have expressed their wish that the
portfolio in its current form continue to be supported by the college’s technology
team…
August 2008
Southeastern Association for Community College Research
29
Stakeholder Perceptions
• CAPR has been well received by stakeholders
(program directors, provost, and members of
the President’s Cabinet)
– “…provides a more representative picture of the
overall quality and sustainability of an individual
academic program.” - SVP, Baccalaureate Programs
and University Partnerships
– “…greatly enhanced SPC’s capacity to evaluate its
lower division academic programs and to
encourage the use of assessment results to
improve College programs.” – VP, Information
Systems, Business Services, Budgets, Planning
August 2008
Southeastern Association for Community College Research
30
Ed Outcomes Site
• To provide a medium for completing the
educational assessment reports as well as
establishing a repository for program specific
information, SPC developed an Educational
Assessment Web site
(https://it.spcollege.edu/edoutcomes/)
• College administration and instructional staff
are provided access to “completed”
assessment reports including the CAPR
• Online access further encourages the use of
assessment data as well as highlighting “best
practices” across the college
August 2008
Southeastern Association for Community College Research
31
Ed Outcomes Site – College Access
August 2008
Southeastern Association for Community College Research
32
Ed Outcomes Site – PD Access
August 2008
Southeastern Association for Community College Research
33
Ed Outcomes Site – PD Access
August 2008
Southeastern Association for Community College Research
34
Ed Outcomes Site – CAPR
• Program Review is just one of the
educational materials included in
the Ed Outcomes site
• CAPRs are uploaded into the site
as a complete document (PDF)
August 2008
Southeastern Association for Community College Research
35
Ed Outcomes Site - CAPR
August 2008
Southeastern Association for Community College Research
36
Ed Outcomes Site – APAR
• Academic Program Assessment
Reports (APARs) are created and
integrated within the Ed
Outcomes site allowing:
– Easier aggregate reporting
– Automated creation of follow-up
reports
– Automated Email approval process
August 2008
Southeastern Association for Community College Research
37
Ed Outcomes Site - APAR
August 2008
Southeastern Association for Community College Research
38
Ed Outcomes Site - APAR
August 2008
Southeastern Association for Community College Research
39
IE Website
August 2008
Southeastern Association for Community College Research
40
Future Direction
 Our ultimate goal is to provide
stakeholders ‘timely’, ‘relevant’,
‘accurate’, and ‘interpretable’ data
through:
 Formatted (dashboard) style reports, and
 On-demand customizable reporting, with
 Valid, reliable, and standardized
measures.
August 2008
Southeastern Association for Community College Research
41
Questions
Department of Institutional
research and Effectiveness
St. Petersburg College
P.O. Box 13489,
St. Petersburg, FL 33733
(727) 341-3059
FAX (727) 341-5411
August 2008
Southeastern Association for Community College Research
42
Southeastern Association
for Community College Research
2008 Conference
Beyond Compliance…Using a
Model of Comprehensive
Academic Program Review to
Improve Education