Transcript Feedback and Evaluation

Experience Examined
Feedback and Evaluation in Clinical
Learning
Catherine Lucey, MD
Goal
• Increase the comfort with which participants
assess, provide feedback to, and complete
evaluations on their learners
“Clinical Learning is Experience
Examined”
Ende and Davidoff, 1992
Two Critical Roles
• Coach: Feedback/ Formative Evaluation
– Information provided during the rotation, describing
performance, with the intent to guide future
performance.
• Judge: Summative Evaluation
– Judgement provided at the close of a rotation assessing
whether the learner met performance standards.
Feedback and Evaluation: Linked but
Different
Feedback
Evaluation
Goal based and objective
Coach
Judge
Formative
Summative
Synchronous
After the Fact
Professional Dev
Quality Assurance
Residents : Help!
80%
How often do
you get?:
70%
60%
50%
Written Fdbk
40%
Verbal Fdbk
30%
Corrective Fdbk
20%
Positive Fdbk
10%
0%
Never/Rare
Often/Freq
Isaacson et al. Cleveland Clinic
Foundation, 1995
Programs : Help!
• 2/3s of Program Directors feel that accuracy
of faculty evaluations are major concerns
• APDIM Precourse, April 1999
• 50% of Program Directors had a recent
troubling feedback/eval session
• APDIM Plenary, October 1999
What are the Barriers?
Brainstorming Session
Barriers:
Fairness, Uncertainty, Time
• Will my evaluation be reliable?
– “ I don’t know what learners today are
supposed to be able to do at this level...”
• Will my evaluation be valid?
– “I didn’t see them do ‘x’ enough to be able to
judge fairly…”
– “I can’t remember …”
Barriers: Sensitivity Concerns
• Will my feedback be well received?
– “what if they cry?”
– “There goes my teaching award….”
• How will my feedback and evaluation be
used?
– “I don’t want to ruin a promising career…”
Overcoming Barriers
• Knowledge
• Expectations of the rotation
• Understanding of the tools
• Attitudes
• Faith in the process and the learner
• Skills
• Targeted observation and
• Constructive Feedback
Road Map for A Successful Rotation
Expectations
Formative Eval
Summative Eval
Case Study
Program:
University of Anywhere
Learner:
Terry, a PGY-1
Preceptor: Dr. Andersen, a general internist in
practice on the eastern shore of Maryland. S/he
knows Terry’s program director from residency.
The program director remembers Dr. Andersen
as a smart doctor and good teacher.
Beginning of the Month
Choose Your Focus
Choose The Right
Tools
Announce your
feedback intentions
1. Choose your Focus: Rotation
Specific Objectives
Knowledge Attitudes
Skills
Medical
Knowledge
Clinical
Judgement
Medical Care
History
Physical
Procedural
Interpersonal
Professionalism
Humanism
Overall Clinical Competence
Examples
• Knowledge:
– Offer a prioritized differential diagnosis for
common ambulatory complaints such as CP,
SOB, Dizziness, Edema…….
• Skills:
– Use and document telephone encounters for dx
and thx follow-up…...
2. Choose Your Evaluation Tools
• Standard Tools For Assessment
– Supervised Direct Patient Care
– Case Presentations
• Standard Tool for Measurement:
– Global Rating Scales
• Behaviorally Anchored Scales
• Developmentally Anchored Scales
BARS *:
The Final Pathway
*Behaviorally Anchored Rating Scales
• ABIM:
123
456
unsatisfactory
satisfactory
789
superior
• Dartmouth
poor
fair
good excellent
Must be
improved
Room for
improvem
Adequate
Skills
Superior Skills
Developmental Scales
USUHS
Northwestern
Reporter
Beginning Difficulty despite
Interpreter
Developing Does with assistance
Manager
Advancing Often does w/o assist
Educator
Capable Usually does w/o assist
Skillful
Always expert, w/o assist
Other Tools to Consider
• Medical Record Audit
– Chart stimulated recall
• Mini CEX
• Standardized Patients
• Nursing and Patient Evaluations
3. Announce Your Feedback
Intentions
• Why will feedback be given?
– Be all you can be...
• When will feedback be given?
– Micro-feedback
– Macro-feedback or Formative Evaluation
• How will feedback be given?
Heart of the Rotation
• Use Micro-feedback
skills to coach the
learner:
– What did they do well?
– Where do they need to
improve?
– What is the next step
for them?
Micro-feedback shapes behavior
•
•
•
•
Timely: Case by Case, 1st Hand
Focused: Choose your pearls
Expected: Part of the culture
Applied: Next step, next patient, goal
directed
• Documented: Progress Notes
How to give effective microfeedback
• Look for behaviors to reinforce as well as
correct
• Be specific about what you saw
• Avoid speculating on intentions
• Use nonjudgmental language
• Encourage self evaluation
• Identify next steps
Reinforcing Feedback
• “I was impressed that you gave Mrs.. Jones plenty
of time to tell her story.” (specific and reinforcing)
• When you are patient like that, you often get a
more accurate history (nonjudgmental and
reinforcing)
• The next step for you is to practice using
clarifying questions for symptoms such as chest
pain.
Corrective Feedback
• “I found a very large liver on Mr. Smith, why do
you think you underestimated its size?” (specific,
encourages self evaluation)
• “When you examine for a liver, it is important to
undress the patient.” (nonjudgmental, forces self
evaluation)
• “As a next step, let’s go over how you position
your hands”
Embarrassment and Humiliation
• Humiliation: Externally driven
– generally negative experience
– avoidable with skilled feedback techniques
• Shame/Embarrassment: Internally driven
– neither entirely avoidable nor entirely negative
Progress Notes
Resident:
Patient:
Date:
Does Well:
Needs Improvement
Next Steps
Feedback Progress Notes
• Action: Document case by case impressions
• Goal: Identify patterns of strengths and
weaknesses in multiple encounters
• Outcome: More specific formative and
summative evaluations
Group Exercise
• Using the scenarios in your handout, role
play micro-feedback and progress note
completion
• Switch roles to allow each individual to
experience the role of feedback provider
Midpoint
Formative Evaluation=
Macro Feedback
Coaching Role Continues
?
Macro-Feedback
• Use patterns identified during microfeedback
• Concretely compare to the agreed upon
goals
• Plan of action to achieve the end goals
– more specific = more achievable
Skill Set: Communicating Evaluations
• Ask: How am I doing as a teacher?
• Ask: What do you see as your strengths;
then weaknesses?
• Tell: Summarize your observations and
ideas
• Ask: What is the take home message, next
steps?
Group Exercise
• Practice the Ask, Ask,Tell, Ask format of
leading an evaluative session
• Check your action plans for specificity
End of the Month
Summative Evaluation:
Did they meet standards?
Met
Exceeded
Failed
Skill Set: Pitfalls in Evaluation
• Range Restriction
• Hawk/Dove
• Halo Effect
Group Exercise
• Complete the Attached evaluation form,
using the notes that you have taken during
the role play sessions.
Conclusions
• Feedback and Evaluation are critical
components of successful clinical learning
• Optimal performance of these skills requires
collaboration between the program and
preceptor
Conclusions
• Less is Frequently More:
– Choose your focus for evaluation
– Provide feedback frequently, in small bits of
manageable information
– Next steps should be measurable
• Feedback is a two way street