Transcript Document

Clinical Skills Testing for High
Stakes
John R. Gimpel, DO, MEd
NBOME
Objectives
• You will be able to describe the clinical
skills components the examinations of the
major North American medical
professions as of 2003
Objectives
• You will be able to discuss exam and case
development issues, psychometric
properties, and standard setting;
differentiating formative and lower
stakes assessment vs. high stakes national
testing
Objectives
• You will be able to describe the
COMLEX-USA-PE, the clinical
skills/performance evaluation component
of COMLEX-USA, appreciating that this
distinctly osteopathic examination will be
the most valid assessment available to
test the clinical skills of osteopathic
physician candidates
Current Status of Performance Based
High-Stakes Testing
*Medical Council of Canada- QE
1992- Required for licensure
*Educational Commission for FMG’s
1998- Required for licensure
*National Board of Medical Examiners
2004-USMLE Part II will include CSE for
Class of 2005 and all IMGs
*National Board of Osteopathic Medical Examiners
2004-2005- COMLEX-USA will likely include PE
as component of Level II
Psychometric Considerations in
Performance Testing
Validity
• Does the assessment provide measure of what it is
supposed to?
Assuring the Validity of
COMLEX-USA-PE Scores
• Case Development
– Sampling
• skills
• content area
– Scoring criteria
• necessary
tasks/questions to
provide patient care
Validity Evidence
•
•
•
•
•
Content
Internal structure
Relationships with criterion measures
Predictive Validity
Consequences
– evaluation drives learning
Reliability
• How consistent are the
examinee scores?
– want to ensure that an examinee’s
observed score is a reasonable
reflection of his/her “true” ability
– Timing features of exam
important (vs. “speeded
performance”)
– minimize errors of measurement
Enhancing Precision
•
•
•
•
Choice of tasks
Raters
Settings
Administration
conditions
• etc.
Assuring the Reliability of
COMLEX-USA-PE Scores
• Case development**
• Define scoring rubrics
• Eliminate extraneous
sources of error
–
–
–
–
training
quality assurance
benchmark videos
performance fidelity
Case Development Issues
• Cases are “vehicles” to measure
skills
–
–
–
–
Who are the “target” examinees?
Specificity
Difficulty
Essential maneuvers and
questions?
– Sampling from domain
– Differences can be major for high
stakes purposes
Standard Setting
• Process used to arrive
at a passing score
– Competent to practice
– Credential
– Master/ non-master
– etc.
Standard Setting
• No widely accepted and
validated standard-setting
methods for use with
performance assessments
– Adaptations of MCQ-based
methods are not completely
satisfactory
– Many “new” techniques look
promising but additional
research is necessary
Standard Setting Issues
• Performance standard setting is a
judgmental process
– There is no “gold standard”
– Sound and defensible procedures can be
adopted
• Performance standards are method
dependent
– Selection of method must be clearly defended
• Performance standards should be
periodically revisited / evaluated
– Cross-validation across judges and students
is needed
Standard Setting Frameworks
• Criterion-Referenced
– Standard defined with regard
to an acceptable specific
measure of performance
•
•
Test-centered (inspection of
items of test components)
Examinee-centered (inspection
of examinee performances)
– Leniency/severity of judges
will affect content-based
performance standards
• Norm-Referenced
– Standard set based on
performance of some
selected group
– Standard will change
as the ability of the
normative sample
changes
– Unknown falsepositive and falsenegative rates
Examinee-Centered
• Contrasting Group
– Form 2 (or more) groups of
people (e.g., masters, nonmasters)
– Look at the 2 score
distributions from the test and
pick the point that maximizes
the probability of correct
decisions
– ECFMG CSA
– USMLE CSE
– COMLEX-USA-PE
• Borderline Group
– Identify sample of
examinees who could be
classified as “borderline”
– Select a point in this
distribution (e.g.,
median) to signify
borderline performance
– MCC Part II
Standard Setting
• In general, examinee
centered standard setting
methods are preferable for
OSCEs
• Enhancements to current
methods, including a focus
on validation of cut-scores,
will help ensure the fairness
of assessment decisions
Comprehensive Osteopathic Medical
Licensing Examination - USA
D
O
National Board of Osteopathic
Medical Examiners
COMLEX – USA - PE
7/17/2015
NBOME-COMLEX-USA-PE
21
Purpose of COMLEX-USA-PE
• Documents the clinical skills expected of
medical students prior to entry into the
first postgraduate training year
• Augments the information provided by
NBOME to state medical licensing
boards to assist them in making more
informed licensure decisions regarding
candidates
COMLEX-USA-PE
TEST DESIGN
•
•
•
•
•
•
12 clinical encounters (13 minutes)
All with SPs
25% involve DO Examiner scoring of OMM
All have SOAP Notes scored by DOs
9 minute post-station- SOAP Note
Directions: “Evaluate and treat the patient as you
see fit”
COMLEX-USA-PE Blueprint
• Osteopathic Medical Practice Axis
• Patient Presentation Axis
• Case Content/Clinical Encounters
Clinical Skills
“Osteopathic Medical Practice” Axis
•
•
•
•
Medical History–Taking
Physical Examination
Doctor-Patient Communication
Patient Management (OMM,
SOAP Notes)
Medical History-Taking
•
•
•
•
•
•
•
•
Eliciting information about the patient’s
present condition
subjective experience of the illness
past medical condition
social history
Lifestyle and stressors
risk factors
mental health
family medical history
Physical Examination
Diagnostic data-gathering involving:
•
•
•
•
•
•
use of observation
palpation (*emphasis appropriate for
D.O.s)
auscultation
percussion
use of diagnostic tools
Doctor-Patient Communication
•
•
•
•
•
Verbal and nonverbal expressions that
promote change
promote rapport
express empathy and respect for the patient
provide information about what is to take
place
educate the patient about his/her condition
and treatment options
Patient Management
Observable behaviors that involve patient care
including:
* written communications (SOAP Note)
differential diagnosis
diagnostic investigations
clinical problem-solving
therapeutic interventions
* osteopathic manipulative treatment
Physician Examiners
• DO examiners are also
used to:
– develop all the cases
– score performance
tasks
(OMM Assessment)
– evaluate written
exercises (SOAP Note
for each case)
Osteopathic School
Involvement
•
•
•
•
•
•
Case Authors
Pilot/ Feasibility Studies
Case Testing
Case Development
Students
SP Trainers
19 Schools
4 Schools
6 Schools
7 Schools
6 Schools
7 Schools
Patient Presentation Axis
•
•
•
•
Acute, Chronic, and HP/DP
Primary Care Settings
Age, Gender, and Ethnic Mix
Reflect actual osteopathic physician
practice and expert input
CONTENT
• In osteopathic medical practice- knowledge is
applied in context, and DOs must have the
skills necessary to work in the context of
patient encounters
• Content of PE based on Dimension I of
COMLEX Blueprint, adapted to reflect actual
osteopathic practice and skills
• Scope of PE focuses on common clinical
scenarios
Implementation
• Likely to be delivered at testing centers,
1-2 initially, strategically located to
minimize cost to candidates and ensure
optimal standardization and reliability
• AY 2004-2005: likely to become
component of COMLEX Level II
• Announcement by September 1, 2003
The 19 Colleges of Osteopathic Medicine
ME

NY
MI

IA


CA


AZ

MO

IL
PA
OH
WV


KY
OK

TX

FL

NJ



Presentation Summary
Clinical Skills Testing for High
Stakes
• Any physician interested in entering
practice in North America within a few
short years will need to pass a high stakes
national clinical skills examination
Presentation Summary
Clinical Skills Testing for High
Stakes
• There are some similarities and many
differences between formative and lower
stakes clinical skills assessments done at
individual programs and schools vs. those
done for high stakes national testing
purposes
Presentation Summary
Clinical Skills Testing for High
Stakes
• NBOME and the osteopathic medical
colleges and the profession have created
COMLEX-USA-PE as a distinctively
osteopathic assessment tool; the most
valid assessment of clinical skills for
osteopathic physician candidates
• [email protected]