Transcript Document
“USMLE Step 2 Clinical Skills” Richard E. Hawkins, MD National Board of Medical Examiners Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Presentation Structure • Overview of USMLE Step 2 CS • Measures to ensure fairness and consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Overview of USMLE Step 2 CS Purpose and Delivery Model Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE: Purpose • Three step examination system primarily designed to support the medical licensing process • Jointly sponsored by – Federation of State Medical Boards (FSMB) – National Board of Medical Examiners (NBME) • Step 2 CS: collaboration with ECFMG Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE: Purpose • Overall purpose – Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patientcentered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE: Purpose • Overall purpose – Assess physician's ability to apply knowledge, concepts, and principles, and to demonstrate fundamental patientcentered skills that constitute the basis of safe and effective patient care Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE: Purpose • Patient-centered Skills – History taking / Physical examination – Communication and Interpersonal Skills – Medical Record Documentation • Important for safe and effective patient care – History/PE Diagnosis and Management – Communication Patient Health Outcomes – Medical Record Errors and Patient Safety Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE Step 2 CS • Enhancement to USMLE Step 2 • USMLE Step 2 – components: – Clinical knowledge (CK) – Clinical skills (CS) • Standard for Step 2 – appropriate for entry into postgraduate training Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE Step 2 CS: Logistics • Regional delivery model – Optimal combination of convenience, cost-efficiency and standardization • Five regional test centers across US • Projected examine volume – 30,000+ / year • Individual center capacity – 3 examinations / day (33 examinees); up to 7 days/week Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Requirements of a High Stakes Performance-based Examination Ensuring Fairness and Consistency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring – Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE Step 2 CS: Content • Step 2 CS Blueprint – Defines content categories – In meeting blueprint specifications, each test form provides: • Adequate sampling of content domain • Comparable content between test forms Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona USMLE Step 2 CS: Content Blueprint Content Categories: – Common and important medical problems / patient presentations – Acuity – Age – Gender – Race / ethnicity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Case Content Cardiovascular Respiratory Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Unclassified / multi-system Case Acuity Acute Subacute / Chronic Test Form Patient age Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 + Patient Gender Male Female Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Step 2 CS: Structure & Tasks • 12 patient encounters • 15 min. for encounter / 10 min. for patient note • Each encounter: – Elicit pertinent history, Perform appropriate physical examination, Communicate effectively – Document: • Findings from the history and physical • Diagnostic impression / Further work-up Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Step 2 CS: Score Components • Integrated Clinical Encounter (ICE) – Data gathering: history and physical exam – Patient note • Communication / Interpersonal Skills (CIS) – Gathering information; sharing information; manner & rapport • Spoken English Proficiency (SEP) – Listener effort, examinee pronunciation / word choice Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Scoring Approach: ICE • Data gathering (Hx / PE) – Dichotomous checklists completed by SPs + • Patient note – Physician raters using holistic methods = • Clinical Process + Clinical Outcome Measure Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Case and Checklist Development • Iterative Process – Involves test committees – Focus on clinical presentation – Checklists “limited” – Encounters with SPs – Appropriateness of content and difficulty • Stepwise progression through pilot and calibration stages – Validation / refinement via review of examinee performance data Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Patient Note • Holistic scoring • Rater training – General policies – Case specific: • Consensus development on key features – Calibration phase – Quality Assurance Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Derivation of Communication / Interpersonal Skills (CIS) Scale • Initial instrument – validated ECFMG scale • Reviewed against national consensus recommendations and commonly used scales Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona CIS Subscales • Data gathering skills – Open-ended questions, transitional statements, not interrupting the patient • Information sharing skills – Responsiveness to patient questions/concerns, provision of counseling when appropriate, avoidance of jargon • Personal manner and rapport – Expression of interest in the impact of the illness, concern for patient comfort and modesty Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Step 2 CS: Scoring • Reporting is Pass or Fail only • Examinees must pass all three subcomponents • Feedback to examinees – Performance report – overall and subcomponent outcomes – Failing examinees only – graphical profiles • Intended to show relative strengths and weaknesses Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Sample Performance Report USMLE Step 2 Clinical Skills P/F Outcome Total Test FAIL ----------------------------------------------------------------------------------------- Examination Subcomponents Integrated Clinical Encounter Pass Fail Communication/Interpersonal Skills Spoken English Proficiency Pass Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Sample Performance Profile Lower Performance Borderline Performance Integrated Clinical Encounter Data Gathering XXXXXXXXXXXXX XXXXXXXXXXXXXXXX Patient Note Communication / Interpersonal Skiills Higher Performance XXXXXXXXXXXXXXXX XXXXXXXXXXXXXX Spoken English Proficiency Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona XXXXXXXXXXXXX Threats to Validity and Reliability • Content and/or tasks not relevant or realistic • Individual test forms vary in content coverage • Scoring methods not appropriate for skills tested • Inconsistency in SP portrayal and scoring – Between cases, across sites, over time • Level of difficulty of cases / exams inconsistent • Standard setting approach appropriate Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Ensuring Fairness and Consistency • Rigorous SP (and SP trainer) training • Meticulous attention to quality assurance • Application of equating procedures Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona SP and SP Trainer Training Standardized Patients • Rigorous training – generic and case specific • Electronic delivery (“Ecase”) of case materials • Sign off process for SPs – Criteria : # portrayals, tests SP Trainers • Adherence to training protocols • “Training academy” for SP trainers Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Quality Assurance • Begins with the SP sign-off process • Monitoring procedures and analyses: – Qualitative (portrayal and scoring accuracy): • Live and video review of SP performances – Quantitative • Score-based analyses • Case level and item level comparisons Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Equating Procedures • Within site – SP-case combination • Between site – Central video review Data gathering Communication and Interpersonal Skills • Patient note rater – case combination Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Standard Setting • USMLE Standard Setting System – Committee assigned responsibility to establish and monitor standards • Decision-making process – Survey of constituent opinion – Standard-setting exercises by independent groups – Examinee performance data and score reliability Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Conclusions • Numerous logistical and psychometric challenges were identified and responses described • In order to ensure exam fairness and consistency: – – – – Sound test and case development practices Intensive SP and SP trainer training Rigorous quality assurance Well-considered equating and standard setting procedures • Supported by ongoing research on reliability and validity Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona Speaker Contact Information Richard E. Hawkins, MD National Board of Medical Examiners 3750 Market Street Philadelphia, PA 19104 215-590-9204 / Fax 215-590-9440 Email: [email protected] Websites: http://www.usmle.org http://www.nbme.org Presented at the 2005 CLEAR Annual Conference September 15-17 Phoenix, Arizona