Transcript A Guide to the Assessment of Professionalism in Trainees

A Guide to the Assessment of
Professionalism in Trainees
Maxine A. Papadakis, M.D.
UCSF
Disclosures

Nothing to
disclose
Objectives
Discuss
Assessment of unprofessional behavior
Determine
Consequences for students who have
not obtained satisfactory
professionalism skills
Professionalism:
What is it?
NBME work on defining behaviors
http://professionalbehaviors.nbme.org/200
8ListofBehaviors.pdf
Teaching Professionalism
Undergraduate
Year 1
Imparting the
Cognitive
Base
Promoting
Self-Reflection
Postgraduate
Year 4
Level of Sophistication
Add Social Contract
Capacity to
Internalize
Professionalism
Cruess, Cruess & Steinert, 2010
Practice
Performance Observations of
Fellow
INPATIENT
•
•
Fellow seems easily distracted on rounds.
Fellow has created a climate of tension, and
the medical student and resident on the
team are intimidated to ask the fellow
questions.
CLINIC
•
•
Fellow needs reminders to fulfill her
assigned patient care tasks.
Fellow leaves clinic before others
What should you do?

How would this be handled at your
institution?
Subsequent Evaluation

SUMMARY COMMENTS
She ..mastered the basics…
 She is very intelligent, and when motivated, she
is able to produce exceptional work.


CONSTRUCTIVE COMMENTS
..put in very little effort
 Often shirked even basic requirements, including
following up on a bone marrow bx…
 ..personality conflict with one of the nurse
practictioners on the service..
 Needs to try to work more effectively with her
team and learn to resolve conflict constructively.

Evaluation of
Professionalism
Methods to Assess
Professionalism









Evaluation forms
Critical incidents
Multisource (360 degree) evaluations
Peer assessment
Evaluation sessions
Patient assessment
P-MEX, OSCEs
NBME Assessment of Professional Behaviors
Physicianship Evaluation Forms
Objective vs subjective but 70% pass subjective-easier when
people see numbers
Summary of Professionalism
Assessment Methods by
Richard and Sylvia Cruess
Evaluation
Knowledge Behavior
Written Tests
Oral Examinations
OSCE
Inter/Gobal Ratings
Direct Observation
Simulations
Portfolios
3600/Peer Evaluations
4+
3
2
1
1
1
2
2
0
1
2
3
4+
2
2
3
[APB web page]
The NBME’s Assessment of
Professional Behaviors Program




Focus: behaviors in GME and FACULTY
Approach: multisource feedback and
web –based training of observers
Purpose: to assess the professional
behaviors that are essential for safe,
effective, and ethical health care
Goal: provide formative feedback that
forms the basis for action
Copyright© 2010 by the National Board of Medical
Examiners® (NBME®). All rights reserved.
Assessment of Professional
Behavior Components




A systematically developed instrument to
assess observable behaviors
A web-based system to collect, track and
collate multisource feedback responses
A source of quantitative and narrative
feedback to learners
An educational program to enhance:
Skill
as observers
Skill as feedback providers
Copyright© 2010 by the National Board of Medical
Examiners® (NBME®). All rights reserved.
Sample Score Report
Copyright© 2010 by the National Board of Medical Examiners® (NBME®). All rights reserved.
Evaluation Sessions


All instructors complete an evaluation
form; 5-point rating scale and comments
All instructors participate in formal
evaluation sessions every 3-4 weeks at
all clerkship sites

Clerkship director provides private
feedback to each student the next day
Evaluation Sessions



Significantly improved detection of unprofessional
behaviors
Professionalism deficiencies 2X more likely to be
detected in inpatient rather than outpatient
rotations
Comparing standard checklists, written comments
and evaluation sessions:
Evaluation sessions significantly improved
detection of unprofessional behavior in both
inpatient and outpatient settings
Hemmer et al. Acad Med; 2000: 75:167
Professionalism MiniEvaluation Exercise
Direct Observation and Peer Evaluation
Eight evaluators rate professionalism of resident
Two faculty
Two peer residents
Two junior residents
Two non-doctor professionals
Good reliability & validity for evaluating professionalism
Richard and Sylvia Cruess
Y Tsugawa et al Medical Education 2009
UCSF Physicianship
Evaluation Forms
CONFIDENTIAL
CONFIDENTIAL
School of Medicine
University of California, San Francisco
STUDENT EVALUATION
Student’s Name (please type)
Course (Department and Course Number)
Faculty Preceptor Name
Quarter, Block and Year
Faculty Preceptor Signature
Location:
Student Grade:
Honors ____
Pass ____
Incomplete ____
E (provisional nonpass) ____
F ____
Reason
______________________________________________________
(Circle the most appropriate)
A. Clinical Knowledge and Skills
1) Fund of Knowledge 1
2
2) History Taking
1
2
3) Physical Exam
1
2
4) Oral Presentation
1
2
5) Record Keeping
1
2
6) Problem Solving
1
2
3
3
3
3
3
3
4
4
4
4
4
4
X
X
X
X
X
X
B. Professional and Personal Attributes
7) Attributes and Responsibility
8) Self-Improvement
9) Patients and Families
10) Relations with Health Care Team
1
1
1
1
2
2
2
2
3
3
3
3
4
4
4
4
X
X
X
X
C. Literature Review 1 2 3 4 X
KEY: 1 = problem 2 = concern 3 = adequate 4 = excellent X = not observed/not applicable
Note that students receiving 1's or 2's in Section B will also receive a "Physicianship" evaluation.
D. Summary Comments (comments suggested for Dean's Letter of Recommendation):
E. Constructive Comments (for use of student and advisor in planning future study; not for use in Dean's Letter):
III. PROFESSIONAL AND PERSONAL ATTRIBUTES
7. PROFESSIONAL ATTRIBUTES AND RESPONSIBILITIES
Cannot be relied upon.
Attendance and
punctuality are erratic.
Student’s whereabouts
often unknown. Needs
prodding frequently. Am
concerned over student’s
commitment.
Needs reminders in the
fulfillment of ward
responsibilities including
patient care. Allows
himself/herself to be too
peripheral to active team
activities and patient care.
Can regularly be relied upon in
fulfilling responsibilities as a member
of the ward team and in the delivery of
patients.
Exceptionally conscientious. Outstanding
in attendance, dependability and
punctuality in team activities and patient
care responsibilities. Makes an extra
effort to be integral team member and
assumes high levels of patient care
responsibilities.
8. SELF IMPROVEMENT AND ADAPTABILITY
Completely unaware of
own inadequacies.
Refuses to consider or
make changes.
Resistant or defensive in
accepting criticism. Makes
those offering suggestions
uncomfortable because of
lack of receptiveness.
Accepts criticism when offered. Makes
an effort to change. Does some
supplemental as well as required reading.
Outstanding in soliciting and
receiving criticism with interest and
grace. Able to effect change. Selfmotivated to expand knowledge.
Intellectually aggressive; makes an
extra effort to learn patients problems;
extensive supplemental reading.
9. RELATIONSHIPS WITH PATIENTS
Often insensitive to
patient’s feelings, needs
and wishes. Lacking
capacity for empathy.
Sometimes has difficulty
establishing rapport with
patients or communication
with them. Not always
comfortable interacting with
others.
Relates well to most patients and family
members. Seems to enjoy patients and
family members and the role as a
developing physician.
Outstanding in putting patients
and/or family members at ease and
appropriately communicating
medical information to them.
Relates well with difficult patients.
Shows empathy, compassion and
respect.
10. INTERPERSONAL RELATIONSHIPS WITH OTHER MEMBERS OF HEALTH CARE TEAM
Insensitive to needs,
feelings and wishes of
health care team
members. Poorly
integrated into the team.
Sometimes has difficulty
relating well to health care
team members.
Relates well to most of the health care team
members. Functions well within the team
structure on the wards.
Outstanding in respecting the
feelings needs and wishes of all
health care team members.
Highly integrated into the team
structure.
Physicianship Evaluation
System

Years 3-4
Course of Action

Physicianship Evaluation Form



Unmet professional responsibility
Diminished relationship with health care team
Remediation







Determine student’s rationale for the actions
Explore the student’s understanding of why the actions are
unprofessional
Refer the student to her mentor
Refer the student to mental health services
Monitor the student’s behavior longitudinally
Forward feeding
Leave of absence
Physicianship Evaluation
System




Years 3-4
Years 1-2
Institutional
Graded academic consequences
Physicianship Evaluation Form
Years 1-2

Reliability & responsibility

Self-improvement & adaptability

Relationships with students, faculty, staff
& patients

Upholding Medical Student Statement of
Principles
Institutional Physicianship
Evaluation Form

Unmet professional responsibility
Communications
USMLE deadlines, immunizations
Financial aid
Social networking

Lack of self-improvement & adaptability
Resistant to feedback, arrogant
Unaware of inadequacies

Diminished relationships with
Administrative faculty & staff
Physicianship Evaluation
Forms 1995-2007
Forms
Students
Years 3-4
Years 1-2
106
35
84
29
TOTAL
141
113
Physicianship Form
Distribution in 59 students
Clerkship/Course
# of Students
Ob-Gyn
21
Psychiatry
9
Pediatrics
5
Medicine
9
Medicine Subinternship
4
Neurology
6
Surgery
1
Family Medicine
7
Other
13
TOTAL
75
Male
13
6
2
8
2
3
1
3
8
46
Female
8
3
3
1
2
3
0
4
5
29
Outcome of 12 students with
2-3 Physicianship Forms
 Mentioned in dean’s letter
 Program notification
 Withdrawal
 Dismissal
 Leave of absence
2
1
1
3
2
Unprofessional Behavior
in Faculty and Residents
Evaluation of Faculty by
Student
I felt that her brusque and often rude
manner was inappropriate. She
came across as belittling and rude to
students, but not to patients. Other
times, she was kind. It was an
emotional rollercoaster. I must
stress that she is fiercely
intelligent…but she is emotionally
labile.
Professionalism as
Competence


Defining specific objectives,
competencies
Identifying where within the curriculum
 they
are delivered
 they are measured
Rewarding Professionalism

Arnold P. Gold Foundation
 Humanism
Honor Society
Peer nomination


Medical School Performance Evaluation
(MSPE)
Faculty rewarding from Respect data
Jordan Cohen 1999
We can measure whether or not our students ‘know’
about professionalism, i.e., whether they can
define the word, list its components, recite its
historical development, and recognize the
contemporary threats to its survival.
But how can we know whether they are
professionals, i.e., whether they are destined—or
at the very least, are likely—as physicians to
behave in accordance with professionalism’s
demanding tenets?
Knowledge, while essential, is not sufficient.
Performance is the key.