Transcript American Board of Surgery Examinations: Can We Identify

American Board of Surgery
Examinations: Can We Identify
Surgery Residency Applicants
Who Will Pass the Examinations
on the First Attempt?
John L. Shellito, M.D., Jacqueline S. Osland, M.D.,
Stephen D. Helmer, Ph.D., Frederic C. Chang, M.D.
Department of Surgery
University of Kansas School of Medicine – Wichita
Introduction: The Importance of
Board Certification
• One of the key demonstrations of the
effectiveness of a residency’s training
program
• ACGME Program Requirements-Surgery
– 65% first attempt pass rate for both the
qualifying and certifying exam
Objective
This study was performed to identify and
evaluate factors which predict successful
completion of both the American Board of
Surgery Qualifying Exam (QE) and
Certifying Exam (CE) on the first attempt
Methods
• Retrospective review
– University of Kansas SOM-Wichita
Surgery Residency Program
– Included residents entering the program from
1990-2001
Pre-Admission Measures
•
•
•
•
•
•
Age and gender
Class rank
AOA status
Honors in third year surgery clerkship
Interview scores
National Board of Medical Examiners (NBME)
Part 1 and Part 2 scores or United States
Medical Licensing Examination (USMLE) Step 1
and Step 2 scores
• Rank list number
Post-admission Measures
• American Board of Surgery In Training
Examination (ABSITE) scores
• Resident awards (Research, Teaching,
Outstanding Senior Resident)
• Faculty evaluations of the 5th year
residents
Statistical Analysis
• Univariate analyses were performed using
Fischer’s exact test for the dichotomous
variables (i.e. gender, AOA status, honors)
• Two-sample Wilcoxon rank-sum test was
used for continuous variables (i.e. scores,
age) and ordinal data
• Logistic regression analysis was
performed on the pre-admission variables
Results
• 77 medical students and residents
– 15 subjects excluded
• 5 NBME or USMLE scores not available
• 8 Left the program
• 2 never attempted the QE (both plastic surgeons)
• 62 subjects remained after exclusions
UKSM-W Surgery Residency
Program 1990-2001
• >90% of our graduates attended medical school
in the Midwest (50% from KU)
– One foreign medical graduate
• 16% (10/62) AOA
• Pass rates
– First time: 77.4%
– Eventual: 96.8%
• 2/62 (plastic surgery)
• Majority entered private practice (73%)
• No difference in first time pass rates over the
study period
Results
• No significant difference based on gender
• 50 (80.7%) were male
• First time pass rate
– 74% for males/ 91.7% for females (p=0.179)
• Residents who successfully passed were
slightly younger at time of application
• 27.1 years vs. 28.6 years (p=0.047)
Effect of Class Rank on First Attempt Pass Rates
Class Rank Group
Number
Percent
Top 1/3rd
22/23
95.7%
Middle 1/3rd
14/22
63.6%
Bottom 1/3rd
3/5
60.0%
P=0.0073
Effect of Step 1 scores on First Attempt Pass Rates
Step 1 Group
Number
Percent
Top 50%
(>200)
36/42
85.7%
Bottom 50%
(<200)
12/20
60.0%
P=0.0247
Effect of Step 2 scores on First Attempt Pass Rates
Step 1 Group
Number
Percent
Top 3 Quartiles
(>186)
44/55
80.0%
Bottom Quartile
(<186)
2/5
40.0%
P=0.0447
Results
• Cumulative scores for step 1 and 2
– >382 (top 2/3rds)
• 81.1% first time pass rate
– <382 (bottom 1/3rd)
• 42.9% first time pass rate
Results
Logistic regression analysis showed Step 1
scores to be the only independent
predictor from the group of pre-admission
factors for first time pass rates
Effect of ABSITE on First Attempt Pass Rates
Number
Percent
>50th percentile
41/50
82.0%
<50th percentile
5/10
50.0% P=0.044
>50th percentile
36/40
90.0%
<50th percentile
12/22
54.5% P=0.002
ABSITE Group
PGY-1
PGY-3
Effect of ABSITE on First Attempt Pass Rates
ABSITE Group
Number
Percent
PGY-4
Top 1/3rd
19/22
86.4%
Middle 1/3rd
23/26
88.5%
Bottom 1/3rd
6/14
42.9%P=0.009
PGY-5
Top 1/3rd
23/24
95.8%
Middle 1/3rd
15/21
71.4%
Bottom 1/3rd
10/17
58.8%P=0.004
Results
• Residents winning an award during
residency had a higher first time pass rate
– 92% versus 67%
• Faculty evaluations of 5th year residents
did not predict board success
Results
• Pre-admission factors that failed to predict
success for first time pass rates
– Honors in 3rd year surgery clerkship
– AOA status
• 10/10 first time pass rate, p=0.061
– Interview score
– Ranking in the match
Discussion
• Clerkship grades
• Honors grade in third year surgery
clerkship
• Dean’s letters
• Letters of recommendations
• Interview Score
• Class rank
• USMLE scores
Discussion
• Xu 1998
– Positive correlation between eventual board
certification status and past academic performance
– Included other surgical specialties in addition to
general surgery
• Silber 2005
– Step 2 score single most important predictor for
eventual OB/GYN board certification
– Graduates from a single medical school, multiple
residency programs
Summary
This is first study to examine factors
predicting first time board certification
rates for general surgery residency
graduates for both the QE and CE
Limitations
• Small sample size
• Bias injected from our selection process
– No information about the success or failure of
applicants who were excluded by our
selection committee
• Single residency training program
Summary
• Academic aptitude by itself does not predict the
multifaceted state of competence of a surgeon
• Professionalism, manual dexterity,
communication abilities, and integrity are not
measured by standardized tests
• Sorting applicants that are above a certain
threshold of intellectual capacity solely by test
scores may not be a good way to obtain the best
residents
Conclusion
Our study indicates a positive
correlation between class rank,
USMLE scores, ABSITE scores and
first time pass rates for ABS
certification