The script concordance test as a measure of clinical reasoning: a national validation study Nouh et al The American Journal of Surgery 2012; 203: 530-534

SCT-Clinical Reasoning national validation study • SCT- prior single center study, differentiated between Jr and Sr SR • To determine validity and reliability, administered on a national level • Test given to 202 Residents - 9 Gen Surg programs across Canada • SCT distinguishes intra-operative decision making skills of Jr. vs Sr. gen surgical residents

SCT

• Based on script theory from cognitive psychology • Uses authentic clinical scenarios to compare a trainee’s judgment skills with those of experts • SCT scores- intended to reflect respondents’ competence, interpreting clinical data under conditions of uncertainty

Script Theory Elements • Clinical encounter: – perceiving features of situation – assessing relevant hypotheses – checking for sx & signs that confirm or refute hypotheses – Using related knowledge to guide investigations + Rx Charlin et al Med Educ 2007: 41,1178-1184

Script Theory Elements • Provides explanations for how events are processed • Fields of computer science and psychology • Organization of experiences in script-like formations – Referred to in future – Understand same or similar new situations

SCT A Practical + Reliable Instrument • 2004 Radiology • 2006 Surgery • 2009 Radiation oncology • • Neurology Pediatric emerg med • • • 2011 Dermatology ENT Preclinical med students • 2012 Geriatrics

SCT Example If you are considering the following investigation . . .

and then you find . . .

you would consider the proposed investigation to be . . .

A V/Q scan to R/O pulmonary embolism CXR Shows areas of collapse ∘ much less useful ∘ slightly less useful ∘ neither less nor more useful ∘ slightly more useful ∘ much more useful

SCT- Natl Study Methods

• Res level of training: post grad yrs 1-5 • Jr = R1-2 • Sr= R3,4,5 • Test development: 9 prgrm dir; scenario,2 6(?)s • Ans key: 5 pt scale, ranging from (-2), comp contra-ind to (+2), comp indic • 153 (?)s

SCT- Natl Study Methods Scoring • Every choice sel by expert→ Credit • 22 Gen Sur experts took test • Credit for each answer was transformed proportionally to get a max score of 1 for the modal experts’ choice on each item • Stat analysis

SCT- Natl Study Methods Results • Scores  a dip in R5 progressively from R1to R4 with • Test differentiated R1+2 from R3+4+5 • Site of training had sig effect on scores

Discussion

• Limitations – No observed drop in variability from R1 to R5 – Choice of scale unsettled – Scores depend on cohort of experts – Influence of generalist vs subspecialist – Item production – Setting of “passing” scores • Advantages – Ease of administration; online – Reliable; valid (?)