Transcript Document

Evaluation of Learners
Effective Techniques and Strategies
Jennie Hsu-Lumetta, M.D.
March 16, 2005
1
7/17/2015
“Clinical Learning is
Experience Examined…”
Ende and Davidoff, 1992
2
7/17/2015
Goals and Objectives
1. Define the purpose of evaluations.
2. Identify the steps to providing effective
evaluations.
3. List some potential barriers/errors to
providing effective evaluation .
3
7/17/2015
What Will Be Covered. . .
1. Definition, Purposes, Types of Evaluation
2. Pre-Evaluation Activities
3. Completing the Evaluation
4. Writing Good Comments
5. Discussing Evaluation
6. RIME Model for Clinical Evaluation
4
7/17/2015
Definition of Evaluation
To determine the value or worth of a
Program,
Product,
Performance
Process
5
7/17/2015
Evaluation of Learners
The process by which the teacher
assesses the learners’
knowledge, skills, and attitudes,
based on criteria related to
educational goals.
6
7/17/2015
Why Evaluate?
Past:
Assess performance
Present: Improve performance
Future: Identify future needs
7
7/17/2015
Uses of Medical Student Evaluations
Use
Purpose
Audience
Past
Assessment
Clerkship
Director
Present
Improvement
Student
Future
Dean’s Letter
Dean
8
7/17/2015
Uses of Resident Evaluations
Use
Past
Purpose
Assessment
Audience
Program
Director
Present
Improvement
Resident
Future
Program
Director’s Letter
Fellowship,
Employers
9
7/17/2015
What to Evaluate?
Attitude
Skills
Knowledge
10
7/17/2015
ASK’s
Attitude, Skills, Knowledge
Competency includes the following:
 Attitude: (as observed via behaviors)
Prepared/Punctuality/Proactive
Follow-up with patient/educational issues
 Skills – context specific
 Knowledge
– Has the information
– Able to analyze and synthesize information
– Able to apply it effectively and efficiently
11
7/17/2015
ACGME Competencies
1. Patient Care
2. Medical Knowledge
3. Practice-Based Learning and
Improvement
4. Interpersonal and Communication
Skills
5. Professionalism
6. Systems-Based Practice
12
7/17/2015
ACGME Competencies
Patient Care that is compassionate, appropriate, and
effective for the treatment of health problems and the
promotion of health
Medical Knowledge about established and evolving
biomedical, clinical, and cognate (e.g. epidemiological
and social-behavioral) sciences and the application of this
knowledge to patient care
13
7/17/2015
ACGME Competencies
Practice-Based Learning and Improvement that
involves investigation and evaluation of their own patient
care, appraisal and assimilation of scientific evidence,
and improvements in patient care
Interpersonal and Communication Skills that result in
effective information exchange and teaming with
patients, their families, and other health professionals
14
7/17/2015
ACGME Competencies
Professionalism, as manifested through a commitment to
carrying out professional responsibilities, adherence to
ethical principles, and sensitivity to a diverse patient
population
Systems-Based Practice, as manifested by actions that
demonstrate an awareness of and responsiveness to the
larger context and system of health care and the ability to
effectively call on system resources to provide care that is
of optimal value
15
7/17/2015
ACGME - Evaluation of Residents
use of dependable measures to assess 6
areas of competency
 mechanisms for providing regular and
timely performance feedback to residents
 a process involving use of assessment
results to achieve progressive
improvements in residents' competence
and performance

16
7/17/2015
Types of Evaluation
1.
2.
Formative Evaluation
Summative Evaluation
17
7/17/2015
Types of Evaluation
1.
Formative Evaluation
 Assess a learner’s progress toward
educational goals
 Throughout the teaching process
 Basis for feedback for the learners and to
make teaching adjustments as needed to
learner to reach learning goals
18
7/17/2015
Types of Evaluation
1.
Summative Evaluation
 Final assessment of a learner
 End of teaching/learning experience
 Used to judge a learner’s mastery of the
educational goals
19
7/17/2015
How to Evaluate?
Assessment/Evaluation

Formative (ongoing)
Feedback
Evaluation

Summative (end of rotation)
20
7/17/2015
Elements of
Effective Assessment
Evaluation Prerequisites
 Making Good Comments
 Aligning Comments with Scaled Questions
 Discussing the Evaluation with the Learner

Source: David Rubin, MD and James Woodruff, MD.
Teaching Skills for the Medical Educator Workshop.
University of Chicago, March 23, 2002
21
7/17/2015
Effective Assessment 1:
Evaluation Prerequisites






Understand expectations of the learner
Understand the evaluation criteria
Understand the survey instrument
Have appropriate expectations
Provide on-going feedback
Provide opportunities for observation
22
7/17/2015
Key components of Evaluation
1.
2.
Observation of the Learners
Questioning
23
7/17/2015
Key components of Evaluation :
Observation of the Learners
The teacher can observe learner
performance for direct
evaluation of knowledge, skills,
and attitudes.
24
7/17/2015
Key components of Evaluation :
The One minute Observation








Explain the purpose of the observation to the
learner
Explain how the observation will occur
Select one skill for observation
Inform the patient of what will take place
Observe for a brief period without interrupting
Leave the room and give immediate feedback
Use the information gain to plan your teaching
Repeat to observe learner’s skills over time
25
7/17/2015
Key components of Evaluation :
Questioning of the Learners
1.
2.
Types of Questions
Levels of Questions
26
7/17/2015
Key components of Evaluation :
Types of Questions
1.
Open-ended
 What?
 How?
 Why?
2.
Closed-ended
 Do?
 Can?
 Have?
27
7/17/2015
Key components of Evaluation :
Level of Questions
1.
2.
3.
Recall
Analysis/Synthesis
Application
28
7/17/2015
Effective Assessment 2:
Completing the Form

Write narrative comments first


Comments are most important!!
Grades are a “blunt instrument”
Dr. Rubin, U of Chicago Dept. of Medicine




Give specific examples
Keep comments to focused behaviors
Connect comments to ratings
Check for internal consistency
29
7/17/2015
Sources of Comments

Direct observation of learner

Written records

Responses to questions

Homework assignments

Interactions with healthcare team, patients

Learner self-evaluation

Feedback/Midpoint evaluation
–
Interventions with the learner
–
Learner’s response to interventions
30
7/17/2015
Effective Assessment 3:
Writing Effective Comments





Strengths and weaknesses
Areas of concern
Suggestions for improvement
Ensure internal consistency
Focus on behaviors not personalities
31
7/17/2015
Effective Assessment 4:
Discuss with Learner







Set a time
Allow enough time for discussion
Have the learner perform a self-assessment
Explain your rationale
Emphasize changes in behavior and performance
that leads to desired competency
Suggest areas for future improvement
Give evaluation in oral and written forms
*Adding feedback element increases the
usefulness of the evaluation and avoids surprises
32
7/17/2015
Avoid rating errors:

The “halo/horn” effect:
– based on reports of prior performance

Restriction range:
– using the same rating for all components of
the evaluation

Rating nonperformance attributes:
– using behavior attributes when rating
nonbehavioral qualities
– Evaluate performance ONLY
33
7/17/2015
RIME Model for Clinical Evaluation

Developed by Dr. Louis Pangaro

Model for student evaluation that was
credible and reliable

Student performance -> Progression of
4 developmental steps

Research shows that it is predictive of
future performance
34
7/17/2015
RIME Model for Clinical Evaluation
4 Stages of Clinical Development:
Reporter
Interpreter
Manager
Educator
35
7/17/2015
RIME Model for Clinical Evaluation

Reporter
– collect and report data (by the M3 year)

Interpreter
– interpret history, PE findings and lab results,
create & prioritize a problem list and
differential diagnosis (by end of M3 year)

Manager
– select the most appropriate diagnostic and
treatment options (by 2nd & 3rd residents)

Educator
– identify and address knowledge gaps
(senior residents)
36
7/17/2015
Proficient
Reporter






Interview & History
Physical examinations
Demonstrates adequate
medical knowledge
Generates adequate
problem list
Demonstrates respect
for patients
Consistently reliable &
honest
Interpreter
•Interprets basic test data
•Integrates medical
knowledge with findings
•Generates several reasonable
explanations
•Can discuss issues & process
of informed consent
37
7/17/2015
Advanced
Manager





Develop diagnostic & therapeutic plans
Makes decisions using benefit/risk analysis
Uses evidence-based approach
Performs as a reliable member of the team
Manages relationships with patients
38
7/17/2015
Outstanding
Educator




Demonstrates leadership
Demonstrates self-directed learning
Exhibits critical reading skills
Demonstrates teaching skills
39
7/17/2015
Summary for Effective Evaluators:
Understand learner expectations
 Understand the evaluation instrument:

– E-value, RIME model
Include thoughtful comments
 Connect comments to rating scales
 Have multiple opportunities for assessing
performance
 Evaluate performance ONLY
 Discuss evaluation with the learner

40
7/17/2015
Special thanks:
 Nancy
F. Barrett
 Carol Packard
41
7/17/2015
Case
You sit down with your student on her last day of the
rotation although you’ve never been comfortable
giving evaluations. Knowing evaluations are
supposed to be balanced, you try to list several
positive things and then list problems. When you
mention that the student’s cardiac patient who spent
an extra day in the hospital because she didn’t track
down a stress test result in time on Friday before the
cardiology department closed, she angrily bursts out
how difficult it is on a team where no one will give
her any help. She also says that you haven’t spent as
much time reviewing write-ups with her as other
residents.
42
7/17/2015