Transcript Early Planning For Self Study

William Carey University
Faculty Development
Staying ahead in an era of
assessment
Robert G. Cuzzolino, Ed.D.
Vice President for Graduate Programs and Planning,
PCOM
Osteopathic Medical
Education in an environment
of planning and assessment
Objectives
Be able to distinguish among mission,
goals and outcomes
 Describe how a continuous quality
improvement program works.
 Organize goals or competencies with
related outcomes data in an
understandable way
 Describe the conditions necessary for
strategic planning to yield organizational or
instructional change.

Objectives, continued

Describe methods of longitudinal analysis

Differentiate between the measurement of
student achievement and the assessment
of institutional or program effectiveness
Mission, Goals,
Competencies, Outcomes

External reviewers focus on mission as the
foundation for institutional assessment and
planning.

Mission statements are just a starting point
in the assessment/planning process chain.




Mission
Goals
Competencies
Student Outcomes

Program Assessment

Planning







Mission demands goals
Goals demand learning objectives
Objectives demand measurement
Measurement demands analysis
Analysis demands planning
Planning demands change
Change demands review
in context of mission
Be kind to your outside reviewers.
 The linkages among institutional goals,
program competencies and outcomes are
not intuitive to most visitors, so why not
map them out?

Institutional
Learning Goal
Academic Program
Competency Group
Outcomes Data
II. Communication
The student acquires
interpersonal and
communications skills that
result in effective
information exchange and
teaming with patients, their
families, and other health
professionals.
Postgraduate assessment
(% above comparative low quartile)
Patient Instruction Skills: 98% (x=3.22/4)
Standardized Patient Communications
Assessment (end of year)
[1-9 point assessment scale:
1= poor, low level/ 9= superior skill]
1st Year Standardized Patient
Communications Assessment (end of year)
Rapport
6.8
Empathy
7.2
Confidence 5.3
Info
7.0
Listen
6.6
Feedback
6.1
PE
7.3
2nd Year Standardized Patient
Communications Assessment (end of year)
Rapport
7.4
Empathy
7.3
Confidence 7.9
Info
8.0
Listen
7.4
Feedback
7.3
PE
8.1
A look at a hypothetical
WCUCOM GCO Grid
On Change

1.3. “The COM must connect its strategic
plan to learning outcomes assessment in
order to continuously improve the
educational quality of its osteopathic
medical education program.”

In other words,
“How does my COM make changes?”
“Have the changes we’ve made been
based on outcomes data?”
Assessment Data / SWOT Analysis /Departmental Plans /Community Input /
Committee Actions/ Accreditor Recommendations/ Budget Requests
Planning
Resource Allocation
Responsibility for
Change
Progress Reporting
Annual Plan Renewal
Outcomes, outcomes
and more outcomes
“Constant testing will no more address the
problems with our educational system than
constantly putting an overweight person
on the scale will cure obesity.”
-Anna Quindlen
Agreeing on the terminology

There is a difference between the
measurement of individual student
achievement and the assessment of
institutional or program effectiveness.

Unfortunately, accreditation standards
often refer to both of these as
“learning assessment” or “outcomes.”

COCA Standard 6.6:
“The COM must define, publish and
implement educational outcomes, based
on its own educational objectives that will
prepare students for osteopathic graduate
medical education.”

Your COM can not measure learning
outcomes if it has not articulated its
learning objectives.

Standard 6.5 Documentation guideline for
proving that your COM has stated learning
objectives and assessment methods:
“Course objectives and course syllabi,
evaluation process for all courses, and
outcomes from the evaluation process.”

Standard 6.6.1
“The COM must establish clinical
competencies and a methodology to
ensure they are being met.”
Learning Assessment Plans

Most institutions have or are developing
learning assessment plans.

These plans tend to focus on the individual
student level and define the expected
competencies, assessment methods,
timing and (sometimes) explain how these
outcomes are reported back to the college.

Learning Assessment Plans are valuable at the
individual student assessment level.

They document the measurement of “learning”
up front for students, and also compel faculty to
clearly define objectives and develop measures
of student achievement - - as a system.

While they are a good basis for developing
objectives and corresponding measures of
learning, assessment plans do not produce
program effectiveness data. They illustrate
how you evaluate students… not the program.
Question for the group

What have you learned from Comlex I
results?
200 Comlex scores mean
nothing in and of themselves.
But if the lowest 50 scores
are your PBL track students,
you have a program
effectiveness issue
When does a set of learning
assessment data help in planning?
When it is aggregated
 When it is augmented by other data
 When it illustrates a trend
 When it is benchmarked or comparative
 When it suggests a course of action, a
change, or the need for a resource.

What learning outcomes data
sources can help inform institutional
change?
 Comlex
scores are a mandated
element.
 Others?
Student clerkship ratings
 Course grade trends
 Standardized patient encounters
 Student survey data
 End of clerkship exams
 Alumni surveys
 AOA, state board data
 Freshman surveys & Graduation Surveys
 Match data and residency selection data
 Certification

Council for Higher Education
Accreditation:

“Effectiveness is a broad concept that
refers to the overall attainment of the
mission and goals of a particular institution
or program. As such, it may embrace
various kinds of behavioral outcomes for
students that go beyond student learning,
such as employment, economic mobility,
and contributions to personal and civic
life…”
Summary on outcomes data

In the self study, we have two needs for
academic performance data:
First – Student Assessment
WCUOM must show that individual
student achievement is regularly
measured against the expected
competencies in a planned manner. It is
also important that we indicate when and
how these expectations and their
assessments are communicated.
Second – Program Evaluation
The self study must coalesce individual
student achievement results into
meaningful program assessment data
sets, and then show that the COM
analyzed the outcomes data and took the
appropriate action. Link curricular,
instructional, organizational, and resource
changes to data.
Example of an institutional effectiveness
process – COMLEX and DO student
success

This brief presentation will share information that
was developed by the PCOM learning resource
committee and shared with the DO Curriculum
Committee and the DO Admissions Committee.

It departs from analysis of grades, clinical
observation ratings and mean Comlex scores
and treats retrospective data to inferential
analysis, with conclusions as to meaning,
but leaves the action plan to the faculty.
Tracking Change

This is most difficult aspect of continuous
quality improvement through strategic
planning.
Longitudinal data

What do you (will you) know about practice
characteristics (geographic distribution,
specialty, residency match, certifications)
of your graduates ?

Think ahead – define your questions before
you graduate a class.
Please rate the graduate, «FIRST_NAME» «LAST_NAME», on each of the following
items by circling the appropriate number. In making the ratings please compare the
intern with all interns you have supervised, not just those in your recent group:
Top
Upper
Lower Bottom
Quarter Middle Middle Quarter
Quarter Quarter
Osteopathic Principles
and Practice, which are
established and incorporated
in the development of skills. ____
____
____
____
Competence in Osteopathic Principles and Practice
40
30
20
10
Std. Dev = .69
M ean = 3.2
N = 65.00
0
1.0
2.0
3.0
4.0
Unable to
Judge
____
Question 17: Diagnostic Skill
Value of Diagnostic Skill
"blank"
"weak"
"strong"
Total
Percent
5.1
.6
94.3
100.0
Valid Percent
5.1
.6
94.3
100.0
Cumulative
Percent
5.1
5.7
100.0
Valid
"blank"
"weak"
"strong"
Total
Frequency
153
71
2297
2521
Percent
6.1
2.8
91.1
100.0
100
90
80
70
Percent
Valid
Frequency
129
14
2378
2521
Preparation of Diagnostic Skill
Value of Diagnostic
Skill
60
50
Preparation of
Diagnostic Skill
40
30
20
10
0
Blank
Weak
General Rating
Strong
Valid Percent
6.1
2.8
91.1
100.0
Cumulative
Percent
6.1
8.9
100.0
Question 13: Understanding the Payor/
Reimbursement System
Value of Understanding the Payor/Reimbursement System
"blank"
"weak"
"strong"
Total
Percent
5.6
14.9
79.5
100.0
Valid Percent
5.6
14.9
79.5
100.0
Cumulative
Percent
5.6
20.5
100.0
Valid
"blank"
"weak"
"strong"
Total
Frequency
162
1890
469
2521
Percent
6.4
75.0
18.6
100.0
90
80
70
Value Understanding the
System
60
Percent
Valid
Frequency
141
375
2005
2521
Preparation of Understanding the Payor/Reimbursement Syaytem
50
Preparation of
Understanding the
System
40
30
20
10
0
Blank
Weak
General Rating
Strong
Valid Percent
6.4
75.0
18.6
100.0
Cumulative
Percent
6.4
81.4
100.0
Strategic Plan

Goal #2: Enhance academic quality, faculty
development, scholarly activity and clinical
service.
k) Pursue state approval of interdisciplinary PhD research degree or pursue partnership with
PhD-granting institution (see Goal 5 a)
l) DO curriculum committee to assess the
adequacy of business in medicine content
and encourage co-curricular presentations
on business, management and medical
economics.
m) Convert geriatric lecture content to on-line format; reformat course to focus on
interactive discussion
Summary on assessment
Demonstrating the connection of learning
outcomes to strategic planning is
important.
Student assessment and program
assessment are not the same thing, but
both are important to the self study, and
both run on their own respective form of
learning outcomes data.
The COCA standards carry an expectation
of longitudinal data.
A
look at the PCOM
strategic plan