Transcript Jonathan Fryer, Noreen Corcoran, Brian George, Ed Wang, Deb DaRosa Northwestern University Feinberg School of Medicine, Chicago Illinois.

Jonathan Fryer, Noreen Corcoran,
Brian George, Ed Wang, Deb DaRosa
Northwestern University Feinberg School of Medicine,
Chicago Illinois
Screening of
Applications
Assessment
of
Candidates
Ranking of
Candidates



1) How effective is the ranking process in
selecting residents who will perform well?
2) How effective is the ranking process in
predicting subsequent resident
performances?
3) How effective are “adjustments” made to
the preliminary rank list in improving the
selection of residents who will perform well?

General surgery categorical resident
recruitment between 2002-2011 inclusive
(n=46).
◦ 4 categorical residents 2002-2004.
◦ 5 categorical residents 2005-2011.

Residents who dropped out (n=1) or who
were recruited after the PGY1 year (n=2) were
excluded from the analyses.

We compared how successful candidates were
ranked during recruitment with their
subsequent performance in our program.


1) USMLE˄ Scores alone
2) Unadjusted Ranking Score (URS˄): based
on sum of 3 assessment scores
◦ Academic Profile (Coordinators)
 Medical school rank, USMLE Step 1, Class Rank, Honors
in Surgery
◦ Program Director Review (PDs)
 Research experience, extracurricular/community
involvement, LORs, Personal Statement, Dean’s Letter
◦ Faculty Interview score (Faculty)
 Averaged for 2 independent faculty interview scores.
˄ Higher score is better

3) Final Adjusted Ranking (FAR˅)
◦ Modification of the preliminary rank list generated
by the URS
◦ Based on additional insights about specific
candidates provided by the resident selection
committee and/or leadership




Endorsements from trusted colleagues
Negative interactions with staff
Concerns raised by residents, coordinators
Other?
˅ Lower is better
Screening
Academic
Profile *
*USMLE included
Assessment
Ranking
URS *
FAR


1) ABSITE˄ percentile alone
2) Resident Evaluation Grade (REG˄)
◦ Semiannual evaluation scores (Letter grade: A-F)
◦ Group discussion and grade assignment based on:
 Clinical Evaluations (360°): faculty, peers, med students,
nurses, patients, etc.
 Compliance: evaluations, case log, duty hour log,
conference attendance, etc.
 ABSITE, Mock Oral, PAME scores.
˄ higher is better

3) Independent Faculty Rating/Ranking
(IFRR˅)
◦ Confidential survey with faculty independently
rating all residents using a 7-point Likert scale and
ranking resident within their PGY1 recruitment
cohorts.
◦ * not part of standard resident evaluation at NU
˅ lower is better


Full IRB approval was obtained.
All resident ranking and performance data
was de-identified after collection and
aggregated to protect resident confidentiality.


Semiannual Resident Evaluation Grades (A-F)
were converted to numerical values (5-0,
respectively) and averaged for analyses.
Data from Individual Faculty Rating/Ranking
surveys were averaged for individual
residents.

Statistical Analyses performed using SAS 9.2
software (Cary, NC).
◦ Associations between ranking and performance
parameters were analyzed using Spearman’s
correlation coefficient.
◦ Comparison of ranking parameters between poor
and satisfactory performance used student t-test.
◦ Differences in performance based on ranking range
were compared using F-test.
Poor Resident
Performance Parameter
# (%)
Drop outs
1 of 46
(2.2%)
ABS QE/CE fail
0 of 13
(0%)
REG < C *
1 of 46
(2.2%)
IFFR > 4.0
6 of 46
(13.0%)
ABSITE < 35% *
12 of 46 (26.1%)
comments
excellent evaluations
Probation
>4.0 = below average
* Occurring at any time during residency training
˄ higher is better
˅ lower is better
Recruitment
Ranking
Resident
Performance
USMLE˄
R*
p
URS˄
R*
FAR˅
p
R*
p
ABSITE (%tile)˄
0.61
<0.0001
0.06
0.6952
0.09
0.5891
PGY1 REG˄
0.12
<0.4087
0.40
0.0058
0.17
0.2597
Overall Grade˄
0.16
<0.2783
0.34
0.0219
0.16
0.2783
Independent
Faculty Rating˅
0.22
<0.1409
0.02
0.9020
-0.12
0.4245
* Spearman correlation coefficient
USMLE˄
Poor Resident
Performance Criteria
*REG ˄< C (n=1)
≥ C (n=46)
URS˄
FAR˅
Mean
SD
Mean
SD
Mean
SD
239.00
-
111.00
-
27.00
-
238.42
15.27
110.44
6.00
18.48
11.83
p value
0.8008
IFRR ˅ ≥ 4.0 (n=6)
232.50
17.44
111.33
5.20
13.14
9.12
239.33
14.76
110.33
6.08
19.52
12.00
< 4.0 (n=41)
0.6041
<0.0001
p value
0.3073
*ABSITE ˄ % < 35 (n=12)
228.92
17.70
110.25
5.93
19.25
13.59
≥ 35 (n=35)
243.48
13.05
110.32
6.37
17.95
11.95
p value
0.0057
0.7023
0.9752
0.2199
0.7625
* Occurring at any time during the residency
USMLE˄
Poor Resident
Performance Criteria
*REG ˄< C (n=1)
≥ C (n=46)
URS˄
FAR˅
Mean
SD
Mean
SD
Mean
SD
239.00
-
111.00
-
27.00
-
238.42
15.27
110.44
6.00
18.48
11.83
p value
0.8008
0.6041
<0.0001
IFRR ˅ ≥ 4.0 (n=6)
232.50
17.44
111.33
5.20
13.14
9.12
< 4.0 (n=41)
239.33
14.76
110.33
6.08
19.52
12.00
p value
0.3073
*ABSITE ˄ % < 35 (n=12)
228.92
17.70
110.25
5.93
19.25
13.59
≥ 35 (n=35)
243.48
13.05
110.32
6.37
17.95
11.95
p value
0.0057
0.7023
0.9752
0.2199
0.7625
* Occurring at any time during the residency
Performance
criteria
ABSITE
(% tile)
Rank range
( % tile)˅
Mean
<15 % (n=14)
Overall REG
IFRR
Mean
SD
Mean
SD
Mean SD
61.69 28.82
4.57
0.55
4.50
0.44
3.10
0.78
15 -30% (n=17) 65.57 21.77
4.29
0.79
4.26
0.64
3.16
0.80
>30% (n=22)
4.00
0.76
4.07
0.56
3.22
0.68
P value
(F-test)
SD
PGY1 REG
66.43 23.73
0.8831
0.1106
0.1338
0.9158
Performance
criteria
ABSITE˄
Rank Range
(% tile)˅
Mean
<15% (n=19)
Overall
REG˄
IFRR˅
Mean
SD
Mean
SD
60.84 26.40
4.63
0.62
4.42
0.50 3.13
0.83
15-30% (n=9)
66.17 22.26
3.81
0.66
3.92
0.61 3.52
0.65
>30% (n=22)
70.00 28.81
4.36
0.71
4.54
0.38 2.70
0.43
P value
(F-test)
SD
PGY1 REG˄
0.6475
0.0025
0.0051
Mean
SD
0.0156
Performance
criteria
ABSITE˄
Rank Range
(% tile)˅
Mean
<15% (n=19)
Overall
REG˄
IFRR˅
Mean
SD
Mean
SD
60.84 26.40
4.63
0.62
4.42
0.50 3.13
0.83
15-30% (n=9)
66.17 22.26
3.81
0.66
3.92
0.61 3.52
0.65
>30% (n=22)
70.00 28.81
4.36
0.71
4.54
0.38 2.70
0.43
P value
(F-test)
SD
PGY1 REG˄
0.6475
0.0025
0.0051
Mean
SD
0.0156




Single center study
URS confounded by USMLE score
REG confounded by ABSITE
No formal faculty orientation for IFRR survey


USMLE scores were predictive of subsequent
ABSITE performance only.
Unadjusted Ranking Scores (URS) were
predictive of subsequent performance based
on resident evaluation grades (REGs), while
Final Adjusted Rankings (FAR) were not.



Our resident selection process has generally been
successful in providing us with residents who
perform well.
Our unadjusted ranking score appears to be a
better predictor of subsequent resident
performance than our final adjusted ranking…
….therefore caution should be exercised when
considering adjustments to the preliminary rank
list, as they may not engender selection of better
performing residents.
Effectively defining a reliable rank list “cutoff”,
beyond which performance will predictably
decrease, may not be possible in our system.
◦ Drop outs: 1 (after PGY1 despite excellent
performance)
◦ ABS exams first try pass rate (n=13):
 100%
◦ Ever with REG <C 100% (i.e. probation): 1
◦ IFR > 4.0 (i.e. below average): 6
◦ Ever with ABSITE scores < 35%tile ever: 12
USMLE
URS
FAR
Poor Resident
Performance
Criteria
Mean
SD
Mean
SD
Mean
SD
REG < C (n=1)
239.00
-
111.00
-
27.96
-
≥C
(n=46)
238.42
15.27
110.44
6.00
18.48
11.83
p value
0.8008
IFR ≥ 4.0
232.50
17.44
111.33
5.20
13.14
9.12
< 4.0
239.33
14.76
110.33
6.08
19.52
12.00
<0.000
1
0.6041
p value
0.3073
ABSITE % <
35
228.92
17.70
110.25
5.93
19.25
13.59
243.48
13.05
110.32
6.37
17.95
11.95
35
≥
0.7023
0.2199

Absolute ranking correlation with resident
performance:
◦ Absite
◦ Semiannual evaluation grade
◦ Faculty survey rating


Absolute ranking (AR): Ranking among entire
candidate group (n= 60-80).
Relative ranking (RR): Ranking among cohort
of successful PGY1 applicants (n=4 or 5).

Within resident cohorts FAR did not correlate
significantly with subsequent:
◦ ABSITE scores (r=0.22; p=0.1760)
◦ Semi-annual evaluation scores (r=0.20; p=0.1987)
◦ Faculty survey cohort rankings (r=0.23;0.1175)

Conversely, USMLE scores exhibited a
significantly positive correlation with
subsequent:
◦ ABSITE scores (r=0.46; p=0.0022),
◦ Semi-annual evaluation scores (r=0.41; p=0.0163)
◦ Faculty cohort rankings (r=0.35; p=0.163)


Resident recruitment involves a formal
evaluation of candidates where a variety of
objective and subjective criteria are used to
rank candidates from best to worst.
Preliminary rank lists are often subsequently
“adjusted” based on additional insights about
the candidates.
Screening of
Applications
Assessment and
Preliminary
Ranking of
Candidates
URS
Final Ranking
of Candidates
FAR