Transcript Evaluation:

Learners' Progress in the Clinical
Setting –
Using “Competencies”, “EPAs”, “Milestones”
and “RIME”
Louis Pangaro, MD, MACP
9 January 2013
Pangaro 2013
1
Further reading



ACE Guidebook for Clerkship Directors
(2013)
Practical Guidebook for the Assessment of
Clinical Competence (Holmboe and Hawkins,
2008)
Frameworks for learner assessment in
medicine, AMEE Guides in Medical Education
(Pangaro, ten Cate, 2013)
Pangaro 2013
2
Situation and context

What: Clinical Rotations:





Clerkships, attachments
Residents, house officers;
(Fellows, subspecialty training)
Where: hospital, clinics, physicians
offices - clinical rotations, “workplace”
Who: Teachers

Faculty; House officers
Pangaro 2013
3
Workshop agenda




What is RIME? What is the problem it’s
trying to solve
Practice
How does RIME relate to competencies,
milestones and EPAs
How does it work?
Pangaro 2013
4
Terminology (jargon!) and how to use it

Competencies
Competence

“RIME”



Milestones
“EPAs”
Pangaro 2013
5
The goal: progressive independence of the learner
Learner
Teacher/
program
Content – Goals
(Patients)
after SFDP
Pangaro 2013
6
UNDERSTANDING
ACTION
Pangaro 2013
7
Enduring Problems
with Teachers’ Evaluations
Accuracy
 Agreement within/across
Teachers
 Inferences about Competence
 Utility


Williams, Klamen, McGaghie, Teach Learn Med, 2003
Pangaro 2013
8
Educational Goals
•Mental Models
•Frameworks
•Words
Curriculum
Assessment/Evaluation
Feedback / Grading
Pangaro 2013
9
Why hasn’t it worked?

What do we struggle with?
Pangaro 2013
10
System barriers to evaluation

Emotional*


Cognitive*


Teacher is not a dispassionate servomechanism; “giving a grade”
goals & objectives may vary; be very elaborate
Resources

teachers and course leaders need time for
training
Pangaro 2013
11
approaches
Analytic – knowledge, skills,
attitude
 Synthetic – roles combining KSA
 Developmental – progress is
explicit

Pangaro 2013
12
R.I.M.E. Scheme:

Reporter

Interpreter

Manager/Educator
Pangaro 2013
13
Grade Distributions Univ. of Utah
Fail Low Pass Pass
HP
Honors
Battistone Acad. Med. , 2001
Pangaro 2013
14
Grade Distributions Univ. of Utah
after RIME methods
“O”
R
I
Pangaro 2013
M
E
15
The RIME rhythm is
familiar:
H&P…………. …….S.0…..
Reporter
Assessment..
….A………… Interpreter
Plan………….
….P…………. Manager/
Educator
Pangaro 2013
16
The rhythm of all productive activity

Observation
Reflection
Action
…further Observation

PDSA, etc etc



Pangaro 2013
17
An exploration of terminology
Words we use to communicate
expectations and values
 The importance of consistency
and acceptance to fairness
 Simplicity – is it important?

Pangaro 2013
18
R.I.M.E. Scheme:

Reporter

Interpreter

Manager/Educator
Pangaro 2013
19
a framework to classify level of function

rudimentary reporting: “My patient
has a fever, cough and a bad rash -
it’s vesicular or pustular ….”

rudimentary interpreting: “I think it
might be due to chicken pox or
herpes.”
Pangaro 2013
20

rudimentary managing:
“I’d
consider a smear of the fluid and
a chest x-ray . We might observe
or treat with acyclovir, but I’m
not sure. I’ll have to look this
up.”
Pangaro 2013
21
Reporter




Takes ownership of working in patient
care and monitoring own patients
Answers “What?” questions
Accurately, reliably gathers and
communicates on one’s own
Takes: knowledge, responsibility,
hard-work, trust
Pangaro 2013
22
“journalists as reporters”








Get the facts right
Gain trust
Maintain confidentiality
Don’t plagiarize
Commutate well
Follow-up leads
Track the story
May be at personal risk
Pangaro 2013
23
The standard



more than simple attendance
(“Observer”)
more than repeater or reciter of others’
work.
consistent, reliable data gathering is
essential and must be directly observed,
documented (DOC)
Pangaro 2013
24
Interpreter

Takes ownership of the “Why?”
questions

Prioritizes, analyzes, synthesizes


Students = reasonable, not “right”
Takes: more knowledge, confidence,
greater independence
Pangaro 2013
25
Manager



Takes ownership of the “How”
questions
Proposes actions and options applied
to individual patient
has maturity, skill and knowledge to
negotiate with patients on plans
Pangaro 2013
26
Educator



Takes ownership of becoming
expert
Poses questions, independently
seeks answers
Shares new knowledge, teaches
others, becomes a leader
Pangaro 2013
27
The rhythm is familiar (intuitive):
Observation.
…….S.0…..
Reporter
Reflection…...
….A………… Interpreter
Action……….
….P…………. Manager/
Educator
Pangaro 2013
28
Construct: experience = RIME progress
70
60
50
40
4th year
6th year
30
interns
20
10
0
Reporter
Interpreter
Manager
Educator
Total RIME
2 simulated encounters rated by two faculty
Tolsgaard, et al Acad Med, 2012
Pangaro 2013
29
3. the “Synthetic”framework



“syn-thetic” -putting the learner
back together
“K S A” are all required, integrated
useful for complex tasks
(functioning in patient care)
Pangaro 2013
30
3. the “Synthetic”framework



developed for descriptive
evaluation for clinicians
terms are a bit less generic
useful to describe progress
toward independence
Pangaro 2013
31
example using RIME: intern
W.O. is an intern who “presents”
 45yo woman with acute lower back pain
 detailed description clinical picture suggestive
of acute lumbar strain
 thorough physical examination
 left-sided Para spinal tenderness L2 – L5
 You ask for the vital signs: blood pressure
130/80, heart rate 80
Pangaro 2013
32
example : intern


while intern is writing up findings in the
patient’s record, you interview and
examine patient.
- She asks: “Doctor, can you take my
blood pressure since no one has?”
at what “RIME” level is this intern?
Pangaro 2013
33
Complimentary Approaches
Synthetic
(“steps”)
(NR)
• Reporter
?
• Interpreter
• Manager/
Educator
•
Analytic
(domains)
•
•
•
Attitude
Skills
Knowledge
Professional(ism) ACGME
Pangaro 12
How does RIME relate?
Pangaro 2013
35
Public Health (Government,
Business, Patients)
Practice/Reimbu
rsement
Regulation (LCME, ACGME, ABMS)
“competencies”
Deans – Chairs – Curriculum Committee
Program/clerkship Directors *
*
Family, Debt,
Culture
Communication
Faculty
in
the Micro-system
*
“RIME”
Student –Resident
Patient
Health/Indepe
ndence
Pangaro 2013
36
Competencies
Pangaro 2013
37
Goals: ACGME Competencies






Medical Knowledge
Interpersonal skills
Professionalism
Patient Care
Practice-based learning
System-based Practice
Pangaro 2013
38
“The habitual and judicious use of
communication, knowledge, technical
skills, clinical reasoning, emotions,
values, and reflection in daily practice
for the benefit of the individual and the
community being served”. (Epstein &
Hundert 2002).
Pangaro 2013
39
Understanding  Action
Reporter/interpreter  Manager/educator
System-based practice
Practice-based Learning
and Improvement
Pangaro 2013
40
Sub-competencies and RIME

See Handout
Pangaro 2013
41
competence


the ability to give to every situation that
a professional might face all that
properly belongs to that situation, and
no more
(Pangaro 2000)
Pangaro 2013
42
EPAs

“entrustable” (entrusted)
professional activities (ten Cate)
Pangaro 2013
43
EPAs and Assessment Plan
1
2
3 Inference of progress
Pangaro 2013
44
What’s the least that the learner
should bring to the situation?



If the learner is a student clerk?
An intern?
A finishing resident?
Pangaro 2013
45
What are Milestones



“Milestones” – observable, level-appropriate
tasks that combine knowledge, skill and
attitudes.
synthesize competencies into observed
tasks,
to benchmark progress toward
independence
Pangaro 2013
46
Moving to “demonstration”
Miller’s Pyramid
Pangaro 2013
47
Trying to capture competence
Competencies
6
Subcompetencies
23
Entrustable
16
activities*
Milestones
115 - 140
* Does not need to be reported in NAS
Pangaro 2013
48
Milestones within RIME
6 months
12 months
12 months
24 months
REPORTER
INTERPRETER
MANAGER
EDUCATOR
Acquire accurate
and relevant
history from the
patient
Synthesize all
available data, ..,
to
define each
patient’s central
clinical problem
With supervision,
manage patients
with common
clinical disorders
Develop a
system to track,
pursue, and
reflect
on clinical
questions
Green et al, 2009
Pangaro 2013
49
Milestones in gathering history
Accurate and relevant history
Data from secondary sources
Obtain historical subtleties
Role model for junior team
members
Months to
achieve
6
9
18
30
After Green , et al. J Grad Med Ed, 2009
Pangaro 2013
50
For most third-year
students, curriculum
should be a requirement
to report and an invitation
to interpret.
Pangaro 2013
51
For most interns,
curriculum should be an
invitation to interpret and
manage.
Pangaro 2013
52
x = proficiency with a patient
M
E
X
X
XX
X
X
XX
I
XX
R
XX
XXX
XXX
2
XX
X
X
X
3
[NOT EMPIRIC DATA]
X
X
X
X
X
x
XX
x
Xx
XXXX
XX
XX
X
X
X
X
4
PGY1
Pangaro 2013
PGY2
PGY3
53
M
E
X
X
I
XX
XX
R
XX
XXX
XXX
2
XX
X
X
X
3
X
X
XX
X
x
XX
x
XX
X
X
X
X
Xx
XXXX
XX
X
X
X
X
X
4
[NOT EMPIRIC DATA]
PGY1
Pangaro 2013
PGY2
PGY3
54
Summary
Why are we here?
Pangaro 2013
55
Medical Knowledge
(Procedural Skills)
Professionalism
Communication Skills
Patient Care
System-based Practice
Practice-based learning & Improvement
Pangaro 2013
56
Further reading



ACE Guidebook for Clerkship Directors
(2013)
Practical Guidebook for the Assessment of
Clinical Competence (Holmboe and Hawkins,
2008)
Frameworks for learner assessment in
medicine, AMEE Guides in Medical Education
(Pangaro, ten Cate, 2013)
Pangaro 2013
57