Transcript The Use of a Surgery Specific Written Examination in the Selection Process of Surgical Residents Daniel Farkas, Kamal Nagpal, Ernesto Curras, Ajay Shah, John.

The Use of a Surgery Specific
Written Examination in the Selection
Process of Surgical Residents
Daniel Farkas, Kamal Nagpal, Ernesto Curras,
Ajay Shah, John Cosgrove
Department of Surgery
Bronx-Lebanon Hospital Center
Albert Einstein College of Medicine
Background


Choosing residents for a surgical residency
program is difficult and time consuming
Programs are looking for many characteristics,
and among them is the ability to acquire surgical
knowledge
Programs are required by the RRC to have 65% of
their residents pass the ABS exams on the first try
 These pass rates are available online to prospective
candidates

Background

Many programs use a candidates USMLE scores
to help identify their ability to gain surgical
knowledge
Some studies have found a correlation between
USMLE scores and ABS exams passing scores
 Some conflicting evidence as to whether step 1 or
step 2 is the better test
 In addition, the USMLE is broad based and not
surgery specific

Study Purpose


The purpose of this study was to look at a
different method of assessing a candidate’s
ability to gain surgical knowledge
Would using a surgery specific written exam
(SSWE) be a better assessment of this ability as
compared with the USMLE?
SSWE

50 question multiple choice SSWE was created



Questions made by a faculty member, using standard
surgical textbooks
Residency candidates were given this exam on
their interview day
Each exam scored out of 100% and added to
candidates folders
Study Design


Retrospective study of residents in our program
between 2004 and 2012
Academic folders were reviewed and preresidency information was collected
USMLE part 1 score (USMLE-1)
 USMLE part 2 score (USMLE-2)
 SSWE score


Intra-residency data collected

ABSITE score in PGY 1 through PGY 5 (ABSITE1 to ABSITE-5)
Study Design



Residents with all three pre-residency scores
were included in the study
Correlation coefficients were calculated between
the pre-residency exam scores and the intraresidency exam scores
P values of less than 0.05 were considered
significant
Results


53 residents had partial information
43 residents had all three pre-residency scores
and were included in the study
38 had an ABSITE-1 score
 24 had an ABSITE-2 score
 18 had an ABSITE-3 score
 17 had an ABSITE-4 score
 14 had an ABSITE-5 score

ABSITE-1 ABSITE-2 ABSITE-3 ABSITE-4 ABSITE-5
n = 38
n = 24
n = 18
n = 17
n = 14
USMLE-1
r = 0.327
p = 0.045*
r = 0.321
p = 0.126
r = 0.346
p = 0.160
r = 0.165
p = 0.527
r = -0.150
p = 0.610
USMLE-2
r = 0.314
p = 0.055
r = 0.187
p = 0.381
r = 0.415
p = 0.086
r = 0.245
p = 0.343
r = 0.044
p = 0.880
SSWE
r = 0.656
p < 0.001*
r = 0.275
p = 0.194
r = -0.179
p = 0.476
r = -0.399
p = 0.113
r = -0.060
p = 0.838
* Statistically significant
ABSITE-1 ABSITE-2 ABSITE-3 ABSITE-4 ABSITE-5
ABSITE-1
ABSITE-2
ABSITE-3
ABSITE-4
* Statistically significant
n = 21
r = 0.573
p = 0.007*
n = 15
r = 0.850
p < 0.001*
n = 17
r = 0.906
p < 0.001*
n = 12
r = 0.593
p = 0.042*
n = 14
r = 0.904
p < 0.001*
n = 15
r = 0.824
p < 0.001*
n=9
r = 0.870
p = 0.02*
n = 11
r = 0.765
p = 0.006*
n = 12
r = 0.682
p = 0.015*
n = 14
r = 0.406
p = 0.150
Results - Summary


USMLE may correlate with ABSITE-1 scores,
but a SSWE has a much stronger correlation
with ABSITE-1 than either step 1 or 2
ABSITE-1 scores correlate significantly with
ABSITE scores in future years
Discussion

Identifying residents that will do well on their
board exams is critical to the success of a
surgical residency program
Part of a resident’s academic achievement is due to
the educational activities within the program
 However, studies have shown that residents with
poor USMLE scores have higher chances of failing
the boards.1,2

1.
2.
Shellito JL, Osland JS, Helmer SD, Chang FC. American Board of Surgery examinations: can we
identify surgery residency applicants and residents who will pass the examinations on the first attempt?
Am J Surg. 2010;199:216-22.
de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery
qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145:852-6.
Discussion


Corneille et al1 showed that a targeted surgery
exam during residency was able to predict which
residents would do well on the ABSITE
Our study showed that by giving this exam to
residency candidates, we could predict which
candidates were more likely to do well on the
ABSITE
Corneille MG, Willis R, Stewart RM, Dent DL. Performance on brief practice examination identifies
residents at risk for poor ABSITE and ABS qualifying examination performance. J Surg Educ. 2011;68:2469.
Limitations

Correlation only significant with ABSITE-1
Possible with a bigger sample size this would change
 ABSITE-1 correlated with ABSITE scores in later
years
 ABSITE scores have been shown to correlate with
board passing rates1


SSWE scores were absolute whereas other
exams were percentile scores

1.
Makes it harder to compare between years
de Virgilio C, Yaghoubian A, Kaji A, et al. Predicting performance on the American Board of Surgery
qualifying and certifying examinations: a multi-institutional study. Arch Surg. 2010;145:852-6.
Limitations

SSWE only useful for surgical knowledge
5 other competencies
 Many other important skills in being a successful
surgical resident and a successful surgeon


Not practical for every program
Many candidates are wooed by other programs on
their interview day
 Good potential candidates could be turned away by
getting an extra exam

Conclusions



Notwithstanding the limitations, the SSWE is a
very strong predictor of who will do well on
ABSITE-1
Higher ABSITE-1 scores is correlated with
higher ABSITE scores in later years
Higher ABSITE scores can predict who will
pass their board exams on the first chance
Final Conclusion

Residency programs looking for residents who
will pass their boards on the first attempt are
more likely to find these residents using a SSWE
than by using the USMLE
Acknowledgements






Kamal Nagpal
Ernesto Curras
Ajay Shah
John Cosgrove
Saundra King
Nancy Rivera