Transcript Chief Residents Quality Improvement Project 2011

Improving Academic Program Quality by
Standardizing and Streamlining the
Evaluation Process for Programs and Faculty
Graduate Medical Education
Stanford University Medical Center
Ann M. Dohn, MA
Designated Institutional Official
Nancy Piro, PhD
Education Specialist/ Program Manager
Department of Graduate Medical Education (GME)
Session Objectives
• The Situation and Problems/Issues: Why Did we Do
This?
• What Did We Do?
• How Did We Do It?
• Results
Department of Graduate Medical Education (GME)
Why Did We Do This?
• The Situation:
– The largest number of citations for our
programs were in the area of evaluations
– Program Directors’ Needs Assessment Results
• One of the Top Five Requests for the Department of
GME = Templates for Evaluation
– GME Review of Evaluations Results
Department of Graduate Medical Education (GME)
What Did We Find?
•
•
•
•
•
Poor quality questions
Inappropriate response scales
Questions not core competency based
Evaluations not even being conducted
Program Director frustration with the whole
process
Department of Graduate Medical Education (GME)
Setting the Stage
• PROGRAM EVALUATION
– 82 Programs
– 14 Citations
What Can We Do???
Department of Graduate Medical Education (GME)
What Did We Do? – STEP 1
• Step 1. Standardized Questions
– Collaboratively developed Prototype Questions
for Programs to Use for Evaluation of Faculty
and Residents
• Core Competency Based
• Unbiased
– Educated Faculty and Program Directors on
Removing Unintended Cognitive Bias from
their Evaluations and Evaluating
Department of Graduate Medical Education (GME)
Tools Were Not Sufficient
• Used group consensus to finalize the questions
• Distributed the questions to the Program Directors
• Educated them on removing bias
but…. the frustration still remained
Department of Graduate Medical Education (GME)
What Could We Do? – STEP 2
Step 2. Centralized the Annual Program Evaluation
Process.
GME now administers program evaluations for all
82 programs
• Used the Standardized Program Evaluation for :
– Faculty
– Residents/Fellows
Department of Graduate Medical Education (GME)
How Is This Implemented?
• GME electronically delivers the evaluations to all
their faculty and residents/fellows…
– Programs can choose one of two dates –
• Feb or May
Department of Graduate Medical Education (GME)
Standardized Program Evaluation by
Faculty Form
Department of Graduate Medical Education (GME)
Standardized Program Evaluation by
Residents/Fellows Form
Department of Graduate Medical Education (GME)
Generate reports for each program:
1) Program Eval by Faculty
Department of Graduate Medical Education (GME)
Generate reports for each program:
2) Program Evaluation by the
Residents/Fellows
Department of Graduate Medical Education (GME)
But still not enough
• Program Directors had the data but didn’t use
it…...
SIGH…..
Department of Graduate Medical Education (GME)
The Total GME Annual Program
Review Package
Aggregate Data Delivered “to you”
1. Evaluation results posted for all Program
Directors on their Residency Management System
website
•
•
•
•
Program Eval by Faculty
Program Eval by Residents/Fellows
Internal GME House Staff Annual Survey
ACGME Survey (if available)
Department of Graduate Medical Education (GME)
The Total GME Annual Program
Review Package
2. Review Checklist – All the essentials that need to
be covered
Department of Graduate Medical Education (GME)
GME Website Support
Department of Graduate Medical Education (GME)
The Total GME Annual Program
Review Package
3. Template Agenda for the Annual Review Meeting
Department of Graduate Medical Education (GME)
The Total GME Annual Program
Review Package
• Sign-In Sheet /Documentation for web
Department of Graduate Medical Education (GME)
Standardized Program Evaluation
• Minutes (Meeting Documentation) Template
Department of Graduate Medical Education (GME)
Standardized Program Evaluation
• As soon as completed, we post the aggregated summary reports
for both Faculty and Residents to our online Residency
Management System for each Program:
Department of Graduate Medical Education (GME)
Standardized Program Evaluation
• Action Plan Documentation Template /Example of an
action plan from one of our Programs….
Department of Graduate Medical Education (GME)
Each program is reviewed by GME
• Ensures all the requirements of the APR are completed and
documented:
• …
Department of Graduate Medical Education (GME)
Outcomes
• 100% compliance in 2010!!
Department of Graduate Medical Education (GME)
Faculty Evaluations
• At the Dean’s request, GME developed
standardized questions for residents and fellows to
evaluate their faculty
Department of Graduate Medical Education (GME)
Annual Academic Evaluation of Faculty
Department of Graduate Medical Education (GME)
Roadblock
• Could not get PD consensus
– Some faculty did not want the results going
directly to the Dean’s office without Chair
‘review’…
Department of Graduate Medical Education (GME)
Where are we?
• Some programs are using it
• The Dean to present to Program Directors
Department of Graduate Medical Education (GME)
Summative Evaluation
• What is summative evaluation?
– A summative evaluation is a method of judging the
competence of a trainee at the end of the program
(summation). The focus is on the outcome.
“Assessment with the primary purpose of
establishing whether or not performance measured at
a single defined point in time meets established
performance standards, permanently recorded in the
form of a grade or score.”
Department of Graduate Medical Education (GME)
Summative Evaluation
• Why do we do this?
– A program director must provide timely
verification (written documentation) of
residency education by completing summative
performance evaluations:
1) for all graduating residents/fellows
2) for residents who leave the program prior to
completion.
Continued…
Department of Graduate Medical Education (GME)
Summative Evaluations
3) ACGME also requires this for
– residents who are transferring into your
program from another program.
“The program director must obtain written or
electronic verification of previous educational
experiences and a summative competency-based
performance evaluation of the transferring
resident/intern.”
Department of Graduate Medical Education (GME)
Summative Evaluations
• GME created Templates for the programs for
Program to use in completing Summative
Evaluations for their trainees graduating or leaving
the program / transferring…
Department of Graduate Medical Education (GME)
Step 2. Continued…
• Provided Faculty/Program Director Education on
the tools and processes
– Reviewed the requirements of Summative
Evaluation…
Department of Graduate Medical Education (GME)
Summative Evaluation
• Why do we do this?
– A program director must provide timely
verification (written documentation) of
residency education by completing summative
performance evaluations:
1) for all graduating residents/fellows
2) for residents who leave the program prior to
completion.
Continued…
Department of Graduate Medical Education (GME)
Summary
• We have found that:
– Standardizing Forms and Developing Templates
• Saves a lot of time for our Programs/PDs
• Improves feedback to the Programs
– Standardizing Program Evaluations in the GME
Office
• Not only saves time for the programs, but assures GME
that there will be no citations for not having program
evaluations
• Our faculty have been engaged and very positive with
development sessions from GME on eliminating bias
from their evaluation questions and response scales
Department of Graduate Medical Education (GME)
Questions?
CONTACT INFORMATION ~
Ann M. Dohn, MA
Designated Institutional Official
[email protected]
Nancy Piro, PhD
Education Specialist/ Program Manager
[email protected]
Department of Graduate Medical Education (GME)
Department of Graduate Medical Education (GME)
Faculty Development Session
EVALUATIONS
How do we eliminate unintended bias
from our evaluation process?
Department of Graduate Medical Education (GME)
38
What is cognitive bias…
• Cognitive bias is distortion in the way we
perceive reality / information.
• Response bias is a type of cognitive bias which
can affect the results of an evaluation if evaluators
answer questions in the way they think they are
designed to be answered, or with a positive or
negative bias toward the fellow being evaluated
Department of Graduate Medical Education (GME)
39
Where does response bias occur?
• Response bias most occurs most often in the
wording of the question.
– Response bias is present when a question
contains a leading phrase or words.
– Response bias can also occur in rating scales.
• Response bias can be in the raters themselves
– Central Tendency
– Halo Effect
– Similarity Effect
Department of Graduate Medical Education (GME)
40
Examples of Question Bias
• Example 1: "I can always talk to my Program
Director about residency related problems."
• Problem: Terms such as "always" and "never" will
bias the response in the opposite direction.
• Result: Data will be skewed.
Department of Graduate Medical Education (GME)
41
Examples of Question Bias
• Example 2: “Career planning resources are
available to me and my program director supports
my professional aspirations."
• Problem: Double-barreled ---resources and
aspirations… Respondents may agree with one
and not the other. Evaluator cannot make
assumptions about which part of the question
respondents were rating.
• Result: Data is useless.
Department of Graduate Medical Education (GME)
42
Examples of Question Bias
• Example 3: "Communication in my program is
good."
• Problem: Question is too broad. If score is less than
100% positive, researcher/evaluator still does not
know what aspect of communication needs
improvement.
• Result: Data is of little or no usefulness.
Department of Graduate Medical Education (GME)
43
Rating Scale Bias
Competence and knowledge in general medicine.
Poor Fair Good Very Good Excellent
The data will be artificially skewed in the positive
direction with this scale because there are far more
(4:1) positive than negative rating options.
Department of Graduate Medical Education (GME)
44
Rater/Evaluator Bias
Response bias can be in the
evaluators themselves
• Central Tendency
• Similarity Effect
• Halo Effect
Department of Graduate Medical Education (GME)
45
Beware the Halo Effect
• The halo effect refers to a cognitive bias whereby the
perception of a particular behavior or trait is
influenced by the perception of the former traits in a
sequence of interpretations.
• Thorndike (1920) was the first to support the halo
effect with empirical research.
– People seem not to think of other individuals in
mixed terms; instead we seem to see each person
as roughly good or roughly bad across all
categories of measurement.
Department of Graduate Medical Education (GME)
46
The Halo Effect and Expectations
• The halo effect is involved in Kelley's
implicit personality theory
– the first traits we recognize in other
people influence our interpretation and
perception of later ones because of our
expectations.
Department of Graduate Medical Education (GME)
47
Halo Effect Extends to Products /
Marketing
The iPod has had positive effects on perceptions of Apple’s other products…
Department of Graduate Medical Education (GME)
48
Could this impact our evaluations here?
Empirical evidence from our HouseStaff….
A question from the most recent GME HouseStaff survey:
“The general feeling in my program is that your ability will
be labeled based on your initial performance.”
Department of Graduate Medical Education (GME)
17.9%
10.3%
Overall Peds Fellows
71.8%
24.4%
15.6%
54.7%
Overall SHC-LPCH
49
Reverse Halo Effect
• A corollary to the halo effect is the
reverse halo effect (devil effect)
– individuals, brands or other things
judged to have a single undesirable
trait are subsequently judged to have
many poor traits, allowing a single
weak point or negative trait to
influence others' perception of the
person, brand or other thing in
general.
Department of Graduate Medical Education (GME)
50
Blind Spots
• In the 1970s, the social psychologist Richard
Nisbett demonstrated that we may have no
awareness of when the halo effect influences us
(Nisbett, R.E. and Wilson, T.D., 1977)
The problem with Blind Spots is that we are blind to them…
Department of Graduate Medical Education (GME)
51