Transcript Physician Documentation

Standardized Letters of Recommendation
Anna H. Messner, MD
Professor & Vice Chair
Program Director
Otolaryngology/Head & Neck Surgery
Stanford University
Application season
Why have letters?
• Review of 966 LOR to General surgery program
• 24% were helpful in evaluation process
Fortune JB. The content and value of letters of recommendation in the resident candidate evaluative process. Curr Surg 2002
Why have letters? The theory.
• Information on past performance which may be a good
predictor of future success
• Unique features
• Insight into typical performance
•
(as opposed to maximal performance)
• Opportunity to explain a less than perfect application
• Opportunity to promote students he/she envisions as
becoming future leaders in field.
Medicine “seems to have taken up permanent residence
along the shores of Lake Wobegon. All of the applicants are
above average—way above.”
All are outstanding
Schneider A. Why you can’t trust letters of recommendation. Chron High Educ 2000
Writing letters- The basics
1. Introduction
2. Relationship to applicant
Length of time he/she has known subject
Context of relationship
Right to view waived
3. Describe subject’s “record,”
Brief summary of subject’s academic performance
Summary of clinical performance
4. Subject’s traits and performance are evaluated and
compared to peers (+ unique features).
5. Summary sentence
OHNS Words/letter
763 letters
•
Mean = 329 (82-1297)
•
Chair of dept (290) vs
others (350) (p<.001)
Letters
•
Messner A, Shimahara E Laryngoscope 2009
What we currently do…
•Look for Standout adjectives (average 2.6 in OHNS letters)
• star, excellent, superb, gifted, extraordinary, outstanding, unique,
exceptional, unparalleled, superstar, finest, exquisite, stellar, sterling,
phenomenal, ideal, exemplary
•Grindstone adjectives:
• Hardworking, resolve, conscientious, dependable, meticulous, thorough,
diligent, dedicated, careful, responsible, reliable, tireless, solid
•“Gets along with others”, “Team player”
•“Compassionate, Caring, Kind, Empathy”
•“Intelligent”
Features of letters
• Leadership potential
• High match rank
• Sense of humor
• Final sentence: Intentionally ambiguous
• Recommend him without hesitation vs recommend highly vs highest
recommendation vs absolute highest recommendation
• Strong recommendation vs very strong recommendation vs strongest
recommendation
• Compare letters from same letter writer
Watch for doubt raisers
•
•
•
•
•
“Technical skills were average.”
“I have discussed alternative career options”
“Made an effort to be an effective team member.”
“Average fund of knowledge”
“His score of 210 accurately reflects his knowledge level.”
Doubt
Raisers
Doubt
raisers
• “It absolutely amazes me
how he can work with
others.”
• “He’s always trying.”
• “I assure you that no
person would be better for
the job.”
Typical Letter analysis: beginning & end
Dear Program Director:
I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have
known Jane for the past 2 years since she first became interested in Otolaryngology.
Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and
History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an
award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of
fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology.
Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of
Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent
knowledge of the head and neck anatomy and diseases that afflict the head and neck.
She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played
an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and
an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to
work with. She is hard working and while somewhat shy Jane is clearly well informed and well read.
In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head
and neck surgery training program.
As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please
feel free to contact me.
Sincerely,
Typical Letter- ignore duplication
Dear Program Director:
I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known
Jane for the past 2 years since she first became interested in Otolaryngology.
Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts
and History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is
an award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives
of fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and
gynecology.
Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of
Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent
knowledge of the head and neck anatomy and diseases that afflict the head and neck.
She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played
an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and
an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to
work with. She is hard working and while somewhat shy Jane is clearly well informed and well read.
In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head
and neck surgery training program.
As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please
feel free to contact me.
Sincerely,
Typical Letter analysis: standouts, grindstone, team player
Dear Program Director:
I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known
Jane for the past 2 years since she first became interested in Otolaryngology.
Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and
History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an
award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of
fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology.
Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of
Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent
knowledge of the head and neck anatomy and diseases that afflict the head and neck.
She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played
an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and
an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to
work with. She is hard working and while somewhat shy Jane is clearly well informed and well read.
In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head
and neck surgery training program.
As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please
feel free to contact me.
Sincerely,
Typical Letter analysis: doubt raisers
Dear Program Director:
I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known
Jane for the past 2 years since she first became interested in Otolaryngology.
Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and
History. She has achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an
award elected annually by current X medical students to the one third year student who contributes significantly to improving the lives of
fellow students. Additionally, Jane obtained honors in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology.
Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of
Medicine. During her rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent
knowledge of the head and neck anatomy and diseases that afflict the head and neck.
She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played
an active role in the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and
an excellent sense of the anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to
work with. She is hard working and while somewhat shy Jane is clearly well informed and well read.
In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head
and neck surgery training program.
As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please
feel free to contact me.
Sincerely,
Typical Letter
Dear Program Director:
I have been asked to submit a letter of recommendation on behalf of Jane Doe who has waived her right to see this letter. I have known Jane for the
past 2 years since she first became interested in Otolaryngology.
Jane is a fourth year medical student at X School of Medicine. She graduated from Y University in 2002 with a Bachelor of Arts and History. She has
achieved several awards during her medical school tenure including “Outstanding Service” in April 2006. This is an award elected annually by current X
medical students to the one third year student who contributes significantly to improving the lives of fellow students. Additionally, Jane obtained honors
in pediatrics, a high pass in surgery, a high pass in obstetrics and gynecology.
Jane rotated on the Head and Neck Surgery Service in the Department of Otolaryngology Head and Neck Surgery at S School of Medicine. During her
rotation Jane performed in an outstanding manner. She is a bright and intelligent student who has an excellent knowledge of the head and neck
anatomy and diseases that afflict the head and neck.
She was actively engaged throughout her rotation in caring for the head and neck cancer patients in the ward. Additionally, she played an active role in
the operating room where she clearly demonstrated an affinity for surgery. She has excellent hand-eye coordination and an excellent sense of the
anatomy. Jane was well liked by her peers, the residents and the attending staff. She is a pleasant person to work with. She is hard working and while
somewhat shy Jane is clearly well informed and well read.
In general I was extremely impressed with this young student who I feel will make an outstanding contribution to any otolaryngology head and neck
surgery training program.
As a result, I would like to submit a very strong letter of recommendation on behalf of Jane Doe. If I can be of further assistance please feel free to
contact me.
Sincerely,
NOTE: nothing unique, interesting, no comparison to peers,
no mention leadership potential, no humor, no match potential
Letter of Minimal Assurance
“I believe that Joe is a strong candidate for a residency position and that he
will ultimately be an excellent otolaryngologist and contributor to medicine.
He is passionately committed to this goal and he is driven by that
commitment. He is a unique person, and one whom you will enjoy meeting
and interviewing.
I would of course welcome a direct phone call regarding this letter.
Thank you for considering Joe for an interview position.”
Current Status
•
•
•
•
Average program: 250 applications x 4 letters = 1000 letters
Average letter = 329 words
Time = 1 min/letter (without analysis)
1000 min = 17 hours.
• Result: Letters often not used to make 1st cut
• Increased reliance on USMLE scores
• And… many not read at all.
Standardized letter of recommendation (SLOR)
• Council of Emergency Medicine Residency Directors
created SLOR
• Successfully used since 1996
Comparison of SLOR & NLOR
• Part I: Compared 20 SLOR and 20 NLOR
• 7- point Likert-type scale
• 4 raters (2 experienced, 2 inexperienced)
• Results
• Interrater reliability of SLOR = 0.97, NLOR 0.78
• Average time to interpret SLOR = 16 sec, NLOR = 90 sec
Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. Acad Emerg Med. 1998 Nov;5(11):1101-4
Comparison of SLOR & NLOR
• Part I: Compared 20 SLOR and 20 NLOR
• 7- point Likert-type scale
• 4 raters (2 experienced, 2 inexperienced)
• Results
• Interrater reliability of SLOR = 0.97, NLOR 0.78
• Average time to interpret SLOR = 16 sec, NLOR = 90 sec
• Part II: 207 SLOR/NLOR pairs
• Results
• 54% given same numeric value, 39% differed by one point, 6% differed by 2
points. Overall correlation was 0.58.
Girzadas DV Jr, Harwood RC, Dearie J, Garrett S. Acad Emerg Med. 1998 Nov;5(11):1101-4
EM SLOR
• 173 surveys of PD organization (70% response rate)
• “Compared with the narrative LOR, please rate the SLOR in
terms of:”
• Ability to discriminate differences between candidates
• Better 75%, Same 18%, Worse 7%
• Ease of reading and incorporating into ranking scheme
• Better 84%, Same 12% Worse 4%
• Ease of completion
• Better 83%, Same 9%, Worse 4%
• Would you like to continue using the SLOR?
• Yes 90%, No 7%
Keim SM, et al. Acad Emerg Med. 1999 Nov;6(11):1141-6
Proposed Standardized Letter of Recommendation
Standardized LOR
Standardized LOR
Qualifications for OHNS. Compare the applicant to other medical students you know.
1. Patient Care: Ability to develop and justify an appropriate differential and a cohesive treatment plan.
Outstanding (top 10%)
Excellent (top 1/3)
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
2. Medical Knowledge: level of general medical knowledge.
Outstanding (top 10%)
Excellent (top 1/3)
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
3. Medical Knowledge: level of medical knowledge in the field of Otolaryngology/Head & Neck Surgery.
Outstanding (top 10%)
Excellent (top 1/3)
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
4. Practice-based learning and improvement: uses evidence based medicine whenever possible
Outstanding (top 10%)
Excellent (top 1/3)
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
Compare to other medical students
5. Professionalism: Work ethic, professional appearance, willingness to assume responsibility.
Outstanding (top 10%)
Excellent (top 1/3 )
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
6. Interpersonal and communication skills: Ability to interact with others on the health care team.
Outstanding (top 10%)
Excellent (top 1/3 )
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
7. Interpersonal and communication skills: Ability to communicate in a caring manner with patients.
Outstanding (top 10%)
Excellent (top 1/3 )
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
Good (lower 1/3)
Unknown
8. Procedural skills: Rate the technical skills of the applicant.
Outstanding (top 10%)
Excellent (top 1/3 )
Very Good (middle 1/3)
Standardized LOR
C.
Assessment compared to other OHNS residency candidates you have recommended over the past 5
academic years:
9. Commitment to Otolaryngology/Head & Neck Surgery. Has carefully thought out this career choice.
Outstanding (top 10%)
Excellent (top 1/3 )
Very Good (middle 1/3)
Good (lower 1/3)
Unknown
10. Commitment to Academic Medicine: Candidate’s likelihood of pursuing a research/academic career
post-residency.
Highly likely
Likely
Likely will not pursue academic career
Unknown
11. Match potential: If the candidate decided he/she would like to match at your institution, how highly
do you estimate he/she will be on your rank list?
Very likely to match
Likely to match
Possible match
Unlikely match
Unknown
Standardized LOR
Standardized LOR
D. Written Comments (please limit to a maximum of ten lines and primarily include any unique features
about the applicant which are not covered in the rest of the application).
Steve rotated on our service and did a great job. He was always prepared for the
operating room. A real team-player.
Signature:
STUDENT HAS WAIVED RIGHT TO SEE THIS LETTER
Date:___________
Standardized LOR
D. Written Comments (please limit to a maximum of ten lines and primarily include any unique features
about the applicant which are not covered in the rest of the application).
David holds a very sincere passion for his research. When he develops an experimental plan he utilizes all
resources to make sure his plan is executed in an effective and efficient manner. When equipment is not
available, he finds a source or collaborator. When the lab does not know or utilize a technique, he learns
it. When a method has not yet been developed, he develops it. I have no doubt that his passion and
ability to work well and effectively with others will take him far.
Signature:
STUDENT HAS WAIVED RIGHT TO SEE THIS LETTER
Date:___________
Standardized LOR
D. Written Comments (please limit to a maximum of ten lines and primarily include any unique features
about the applicant which are not covered in the rest of the application).
After obtaining his cochlear implant and taking time off to do research, I think John is uniquely positioned
to become a superb otolaryngologist. I don’t believe there is anybody on the planet who is more motivated
than John to do this. Certainly, no one has more insight into the problems associated with being deaf than
John. I rate him in the top 5% of all medical students with whom I have worked during the past 20 years.
Signature:
STUDENT HAS WAIVED RIGHT TO SEE THIS LETTER
Date:___________
Standardized LOR
D. Written Comments (please limit to a maximum of ten lines and primarily include any unique features
about the applicant which are not covered in the rest of the application).
As you know, John did not match last year. He has examined the myriad of possibilities as to why this
happened. While I certainly don’t want to focus this letter on the fact, I can tell you with absolute certainty
that John has gleaned every possible learning opportunity from this experience. With something as
complicated as residency selection, it is fair to say that there rarely is one reason why someone does or
does not match. Although he was appropriately and understandably deeply disappointed, he was able to
both grow from this experience as well as to use his time this year in the most productive way. Sometimes
the measure of a person is best taken during times of adversity. John has shown exceptional maturity,
openness, self-reflection, and commitment throughout.
Signature:
STUDENT HAS WAIVED RIGHT TO SEE THIS LETTER
Date:___________
Advantages of SLOR
•
•
•
•
More consistent information
Less duplication- of contents of application and letters
Less time to complete & interpret
Information less dependent on letter writer (more fair)
• (A superb letter writer may portray a “good” candidate in stratospheric
terms, whereas a mediocre letter writer may portray a “superb”
candidate less positively)
• Still have opportunity to tell about unique features of
applicant
Disadvantages of SLOR
•
•
•
•
Potential loss of nuances
Much harder to make the average student look stupendous
Potential attending MD guilt
Don’t get to show off fancy Department stationary
• But OK to add Logo, or put form on stationary
Next Steps
• Open for comments until February 1, 2012
• Send to [email protected]
• OPDO council will review all comments and revise proposed SLOR
• Assuming response is positive:
• Will post SLOR on SUO-AADO website
• Email distribution to PDs and Chairs
• Need Program director assistance to get the word out to faculty and
applicants
• Investigate information re SLOR in Oto journal
• Start using in spring/summer 2012
• Survey SUO/AADO/PD re use, then revise.
Conclusions
• Traditional letters have the potential to help with identifying
top residency candidates– but often do not.
• A SLOR has the potential to increase the value of LOR
while saving time.
• Please send all comments to:
[email protected]