Transcript Resident Assessment Does the OITE mean anything?

Steven Kane, MD
David Robinson, MD
Atlanta Medical Center












Core Competencies
Prior USMLEs
Interviews
Faculty Evaluations
Intra-operative observation
Clinical observation
Surgical Skills lab
Peer to Peer
Pre medical experience
Ancillary staff evaluations
OITE
Milestones
9.63

“I shall not today attempt
further to define the kinds
of material I understand to
be embraced within that
shorthand description and
perhaps I could never
succeed in intelligibly doing
so. But I know it when I
see it……” 1964
Potter Stewart



Carmichael, Kelly D. MD; Westmoreland,
James B. MD; Thomas, John A. MD; Patterson,
Rita M. PhD
May 2005, Southern Medical Journal
The only statistically significant correlations to
OITE scores were USMLE step I performance
(220) and marital status (married)





Paul J. Dougherty MD; Norman Walter MD; Peter
Schilling MD; Soheil Najibi MD, PhD; and Harry
Herkowitz MD
March 2010, Clinical Orthopaedics and Related Research
Those who averaged in the 27th percentile or lower on
the OITE had a 57% chance of failing the ABOS Part I
examination
Correlation coefficient of 0.53 between the USMLE Step
1 and ABOS Part I
Average correlation coefficient of 0.50 for postgraduate
years (PGY) 2 through 5 OITE scores and ABOS Part I





KLEIN, GREGG R. MD; AUSTIN, MATTHEW S. MD;
RANDOLPH, SUSAN; SHARKEY, PETER F. MD;
HILIBRAND, ALAN S. MD
May 2004, The Journal of Bone and Joint Surgery-American
Volume
Correlation coefficient for the USMLE Step-I score and the
ABOS Part-I percentile score was 0.38 (p = 0.002)
Correlation coefficients were also calculated for the OITE
percentile scores for years 3 (r = 0.52, p < 0.0001), 4 (r = 0.49,
p < 0.0001), and 5 (r = 0.69, p < 0.0001).
There was a high risk of failure (63%; five of eight failed) on
the ABOS Part-I examination when a resident scored below
the twenty-ninth percentile for PGY-3 and below the
twentieth percentile for PGY-5


5 programs evaluated
176 current and former residents evaluated
Chairs or PD’s asked to rate overall
competence and surgical competence along
with the 6 core competencies while
“modestly blinded” to USMLE, OITE, and
ABOS percentiles
A statistical cross
comparison of
USMLE 1, USMLE 2,
OITE 1-5 percentiles,
6 Core
Competencies, ABOS
percentiles, Surgical
Competency and
Overall Competency
upon graduation
was compiled.
Survey / Score Category
Medical Knowledge
OITE %tile YIT-5
OITE %tile YIT-2
Surgical Competency
Medical Knowledge
Overall Competency
Medical Knowledge
Medical Knowledge
OITE %tile YIT-3
USMLE Step 2
PBL & Improvement
Systems-based Practice
Professionalism
Interper. & Comm. Skills
Surgical Competency
Interper. & Comm. Skills
Medical Knowledge
Surgical Competency
OITE %tile YIT-4
Overall Competency
Overall Competency
Medical Knowledge
Patient Care
PBL & Improvement
OITE %tile YIT-2
Systems-based Practice
Patient Care
Surgical Competency
PBL & Improvement
PBL & Improvement
PBL & Improvement
Surgical Competency
Overall Competency
OITE %tile YIT-1
Overall Competency
Professionalism
Overall Competency
Surgical Competency
Systems-based Practice
Systems-based Practice
PBL & Improvement
Professionalism
Overall Competency
PBL & Improvement
OITE %tile YIT-2
Systems-based Practice
Professionalism
Patient Care
Systems-based Practice
Professionalism
Patient Care
USMLE Step 1
Interper. & Comm. Skills
Overall Competency
Professionalism
PBL & Improvement
Professionalism
Patient Care
Interper. & Comm. Skills
Interper. & Comm. Skills
Medical Knowledge
Systems-based Practice
OITE %tile YIT-1
Surgical Competency
Patient Care
Medical Knowledge
Patient Care
Interper. & Comm. Skills
OITE %tile YIT-4
OITE %tile YIT-1
OITE %tile YIT-3
Interper. & Comm. Skills
Surgical Competency
OITE %tile YIT-4
OITE %tile YIT-5
Systems-based Practice
Interper. & Comm. Skills
Patient Care
Professionalism
OITE %tile YIT-3
OITE %tile YIT-5
Score Category
ABOS %tile
ABOS %tile
ABOS %tile
ABOS %tile
OITE %tile YIT-2
ABOS %tile
OITE %tile YIT-3
OITE %tile YIT-5
ABOS %tile
ABOS %tile
ABOS %tile
ABOS %tile
USMLE Step 2
USMLE Step 2
OITE %tile YIT-2
ABOS %tile
OITE %tile YIT-4
OITE %tile YIT-3
ABOS %tile
OITE %tile YIT-3
OITE %tile YIT-2
OITE %tile YIT-1
ABOS %tile
OITE %tile YIT-3
USMLE Step 2
USMLE Step 2
USMLE Step 2
OITE %tile YIT-5
OITE %tile YIT-2
OITE %tile YIT-1
USMLE Step 2
OITE %tile YIT-1
USMLE Step 2
USMLE Step 2
OITE %tile YIT-1
OITE %tile YIT-1
OITE %tile YIT-5
OITE %tile YIT-4
OITE %tile YIT-3
OITE %tile YIT-2
OITE %tile YIT-5
OITE %tile YIT-4
OITE %tile YIT-4
OITE %tile YIT-4
USMLE Step 1
USMLE Step 1
OITE %tile YIT-2
USMLE Step 1
OITE %tile YIT-5
OITE %tile YIT-3
OITE %tile YIT-3
ABOS %tile
OITE %tile YIT-5
USMLE Step 1
OITE %tile YIT-5
USMLE Step 1
USMLE Step 1
OITE %tile YIT-2
USMLE Step 1
OITE %tile YIT-4
USMLE Step 1
OITE %tile YIT-1
ABOS %tile
USMLE Step 1
OITE %tile YIT-1
USMLE Step 2
OITE %tile YIT-5
OITE %tile YIT-3
USMLE Step 2
USMLE Step 1
USMLE Step 1
OITE %tile YIT-2
USMLE Step 2
USMLE Step 1
USMLE Step 1
OITE %tile YIT-4
OITE %tile YIT-1
OITE %tile YIT-4
ABOS %tile
USMLE Step 2
USMLE Step 2
R
0.548
0.451
0.423
0.390
0.385
0.370
0.363
0.337
0.317
0.299
0.298
0.290
0.289
0.288
0.281
0.272
0.260
0.259
0.252
0.234
0.231
0.226
0.218
0.217
0.215
0.203
0.195
0.192
0.183
0.169
0.156
0.155
0.146
0.143
0.136
0.133
0.133
0.131
0.124
0.123
0.116
0.114
0.104
0.101
0.101
0.100
0.098
0.093
0.091
0.088
0.087
0.082
0.079
0.078
0.076
0.073
0.071
0.070
0.068
0.068
0.067
0.066
0.063
0.053
0.053
0.049
0.046
0.046
0.033
0.030
0.027
0.025
0.018
0.015
0.004
0.00
0.00
-0.02
-0.02
-0.04
-0.14
S
65
60
65
65
156
65
154
117
65
23
65
65
53
53
156
65
137
154
62
154
156
89
65
154
53
53
53
117
156
89
53
89
53
27
89
89
117
137
154
156
117
137
137
137
131
131
156
131
117
154
154
63
117
131
117
131
131
156
131
137
131
89
23
131
89
53
117
154
43
60
123
156
53
104
94
137
89
137
65
51
37
p - Value
0.000002
0.000303
0.000446
0.001321
0.000001
0.002415
0.000004
0.000199
0.010091
0.166287
0.015810
0.019020
0.035895
0.036435
0.000382
0.028624
0.002153
0.001167
0.048205
0.003492
0.003719
0.033143
0.080584
0.006906
0.122165
0.144316
0.162399
0.038341
0.021871
0.113589
0.265180
0.146400
0.296710
0.478140
0.202862
0.213215
0.154008
0.125677
0.125109
0.124724
0.212999
0.183033
0.224354
0.239368
0.253204
0.257931
0.225420
0.290990
0.327312
0.275292
0.285162
0.524467
0.395625
0.373473
0.415775
0.406482
0.422821
0.385688
0.437785
0.427352
0.446974
0.537830
0.776877
0.547163
0.623990
0.725595
0.622756
0.573154
0.833658
0.820769
0.765552
0.761063
0.900569
0.876059
0.966199
0.993523
0.971065
0.847326
0.845774
0.799276
0.396155
Variable
Spearman
by Variable N ρ
Prob>│ρ│
ABOS Part 1 Percentile
OITE 1
25
0.170
0.0417
ABOS Part 1 Percentile
OITE 2
65
0.478
<.0001
ABOS Part 1 Percentile
OITE 3
65
0.296
0.017
ABOS Part 1 Percentile
OITE 4
62
0.251
0.049
ABOS Part 1 Percentile
OITE 5
60
0.463
<.0001
Correlation with ABOS
Spearman ρ
Prob>│ρ│
Survey Category
N
Patient Care
65
0.18
0.1501
Medical Knowledge
65
0.54
<.0001
Practice-Based Learning and Improvement
65
0.28
0.0261
Interpersonal and Communication Skills
65
0.27
0.0300
Professionalism
64
0.28
0.0266
Systems-Based Practice
65
0.27
0.0283
Surgical Competency
65
0.37
0.0026
Overall Competency
65
0.36
0.0035
ABOS was more correlative with surgical and overall
competency than OITE (medium correlation)
Survey/Score Category
Score Category
R
S p - Value
Medical Knowledge
ABOS %tile
0.548
65 0.000002
OITE %tile YIT-5
ABOS %tile
0.451
60 0.000303
OITE %tile YIT-2
ABOS %tile
0.423
65 0.000446
Surgical Competency
ABOS %tile
0.390
65 0.001321
Medical Knowledge
OITE %tile YIT-2
0.385 156 0.000001
Overall Competency
ABOS %tile
0.370
Medical Knowledge
OITE %tile YIT-3
0.363 154 0.000004
Medical Knowledge
OITE %tile YIT-5
0.337 117 0.000199
OTIE %tile YIT-3
ABOS %tile
0.317
65 0.010091
USMLE Step 2
ABOS %tile
0.299
23 0.166287
65 0.002415
OITE YIT- 4 Appears to be a throw
away year
Survey/Score Category
Score Category
R
S
p - Value
PBL & Improvement
ABOS %tile
0.298
65
0.015810
Systems-Based Practice
ABOS %tile
0.290
65
0.019020
Professionalism
USMLE Step 2
0.298
53
0.035895
Interper. & Comm. Skills
USMLE Step 2
0.288
53
0.036435
Surgical Competency
OITE %tile YIT-2 0.281
156
0.000382
Interper. & Comm. Skills
ABOS %tile
0.272
65
0.028624
Medical Knowledge
OITE %tile YIT-4 0.260
137
0.002153
Surgical Competency
OITE %tile YIT-3 0.252
62
0.048205
OITE %tile YIT-4
ABOS %tile
0.252
62
0.048205
Overall Competency
OITE %tile YIT-3 0.234
154
0.003492
The only objective score to correlate with surgical
competency was ABOS I percentile (medium correlation)
Survey/Score Category
Score Category
R
S
p - Value
Surgical Competency
OITE %tile YIT-1
0.155
89
0.1464
Surgical Competency
OITE %tile YIT-2
0.281
156
0.000382
Surgical Competency
OITE %tile YIT-3
0.259
154
0.001167
Surgical Competency
OITE %tile YIT-4
0.131
137
0.125677
Surgical Competency
OITE %tile YIT-5
0.192
117
0.038341
Surgical Competency
USMLE Step 1
0.053
131
0.547163
Surgical Competency
USMLE Step 2
0.018
53
0.900569
Surgical Competency
ABOS %tile
0.39
65
0.001321
Overall Competency has no
correlation with OITE YIT 1-5
Survey/Score Category
Score Category
R
S
p - Value
Overall Competency
OITE %tile YIT-1
0.136
89 0.202862
Overall Competency
OITE %tile YIT-2
0.231
156 0.003719
Overall Competency
OITE %tile YIT-3
0.234
154 0.003492
Overall Competency
OITE %tile YIT-4
0.104
137 0.224354
Overall Competency
OITE %tile YIT-5
0.133
117 0.154008


Our “mildly blinded” study of 5 programs,
objective measurements have very little correlation
with a program director or chair’s opinion of a
resident’s overall competency, core competencies
outside of medical knowledge or surgical
competency upon graduation.
The one exception is the ABOS percentile and
surgical competency which is mildly correlative.
“When given a choice on who will
operate on me or my children, with
all other qualities being equal, I
would choose the physician with
the higher scores….”
Steven Frick. June 2012