Transcript Teal

“Show Me How to Get
Past MCQs:
Emerging Opportunities in
Measurement ”
Carol O’Byrne, PEBC
Karen S. Flint and Jaime Walla, AMP
Drs. Frank Hideg, Paul Townsend, &
Mark Christensen, NBCE
Alison Cooper, CAPR
Lila Quero-Munoz, Consultant
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Goals
1.
2.
3.
4.
5.
6.
Gain an overview of performance assessment
Observe and try out electronic & standardized
patient simulations
Consider exam development, implementation
and administration issues
Consider validity questions & research needs
Create computer-administered & standardized
patient simulations with scoring rubrics
Set passing standards
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Part 1 - Presentations
Introduction to performance assessment
– Purposes and objectives
– Models
– Issues, successes and challenges
15-minute presentations
– Four models, including their unique
aspects with two participatory
demonstrations
– Developmental and ongoing validity issues and
research studies
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Part 2 - Break-out Sessions
1. Identify steps in development and
implementation of a new performance
assessment and develop a new station
2. Create a new electronic simulation and set
passing standards
3. Create a new standardized patient simulation
and scoring rubrics
4. Participate in a standard setting exercise using
the ‘Competence Standard Setting Method’
and all the while, ask the ‘hard questions’
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Performance Assessment WHY?
To assess important problem solving,
critical thinking, communications,
hands-on and other complex skills that:
– Impact clients' safety and welfare if not
performed adequately and
– Are difficult to assess in a multiple choice
question format
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
HOW?
• ‘Pot luck’ direct observation (e.g., medical
rounds, clerkships and internships)
• Semi-structured assessments (e.g. orals
and Patient Management Problems)
• Objective, Structured Clinical Examinations
(OSCEs) (combining standardized client
interactions with other formats)
• Other standardized simulations (e.g., airline
pilots' simulators)
• Electronic simulations (e.g., real estate,
respiratory care, architecture)
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Does it really work?
Links in the Chain of Evidence to Support the
Validity of Examination Results:
ℴ
ℴ
ℴ
ℴ
ℴ
ℴ
ℴ
ℴ
Job Analysis
Test Specifications
Item Writing
Examination Construction
Standard Setting
Test Administration
Scoring
Reporting Test Results
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
PEBC Qualifying Examination
• Based on national
competencies
• Two parts:
MCE & OSCE
• Must pass both to be
eligible for pharmacist
licensure in Canada
• Offered spring and fall in
multiple locations
• 1400+ candidates/year
• $1350 CDN
• 15-station OSCE
– 12 client interactions (SP
or SHP) + 3 non-client
stations
– 7 minute stations
• One expert examiner
• Checklist to document
performance
• Holistic ratings to score
exam
• Standard Setting
• Reports – results and
feedback
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Competencies Assessed by PEBC’s
MCE and OSCE
COMPETENCIES
MCE OSCE
%
%
56.5
29
2: Assume ethical, legal and professional
responsibilities
8.5
9
3: Access, retrieve, evaluate and
disseminate relevant information
6.5
5
4: Communicate and educate effectively
1.5
43
23
9
4
5
1. Practise pharmaceutical care
5: Manage drug distribution
6: Apply practice management knowledge
and skills
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Comparing PEBC’s OSCE (PS04)
and MCE (QS04) Scores
Paired Samples Statistics
Pair
1
PS04
QS04
Mean
473.7354
481.2897
N
711
711
Std. Deviation
95.14097
96.62295
Std. Error
Mean
3.56807
3.62364
Paired Samples Correlations
N
Pair 1
PS04 & QS04
711
Correlation
.607
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Sig.
.000
Comparing PEBC’s OSCE and
MCE scores
Paired Samples Statistics
OriginS04
1
Pair
1
2
Pair
1
USA
Pair
1
PS04
QS04
PS04
QS04
PS04
QS04
Mean
500.0402
502.4589
353.7158
387.7982
445.4000
448.9459
N
560
560
114
114
37
37
Std. Deviation
68.96394
85.94416
109.35265
88.43879
99.66310
95.69075
Std. Error
Mean
2.91426
3.63180
10.24181
8.28305
16.38451
15.73146
Paired Samples Correlations
OriginS04
1
2
USA
N
Pair 1
Pair 1
Pair 1
PS04 & QS04
PS04 & QS04
PS04 & QS04
560
114
37
Correlation
.445
.571
.688
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Sig.
.000
.000
.000
Holistic Rating Scales
COMMUNICATION Skills (1)
100
80
Percent
– Rapport
– Organization
– Verbal and nonverbal expression
Holistic Scale Means
Problem-solving OUTCOME (2)
0
1
2
Holistic Scale
3
Analytical Score Means
Overall PERFORMANCE (3)
80
Percent
Comm & Outcome
Thoroughness (checklist)
Accuracy (misinformation)
Risk
40
20
– Information processing
– Decision making
– Follow-up
–
–
–
–
60
60
40
20
0
1
2
3
4
5
Competencies
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
6
PEBC QUALIFYING EXAMINATION - PART II (OSCE)
EXAMINATION FEEDBACK REPORT
Candidate ID
Table 1
Status: Fail
RATINGS
Communication
Your average 3.10
Group average 3.67
Rating
4=Acceptable
3=Marginally acceptable
2=Marginally unacceptable
1=Unacceptable
Outcomes
Your average 2.33
Group average 2.90
Rating
# of Stations (of 15)
4=Problem solved
2
3=Solved marginally
5
2=Uncertain/marginally unsolved
4
1=Problem unsolved
4
Overall Performance
Your average 2.51
Group average 3.01
Rating
4=Acceptable
3=Marginally acceptable
2=Marginally unacceptable
1=Unacceptable
Table 2
# of Stations (of 15)
3
5
4
3
MISINFORMATION AND RISK/INEFFECTIVE THERAPY
Misinformation
Risk or ineffective therapy
Table 3
# of Stations (of 12)
4
6
1
1
# of Stations/Instances (of 15)
3
2
Group Average
1.31
0.63
COMPETENCY SCORES
Competency
Your Score (%)
1 Practise pharmaceutical care
47
2 Assume ethical, legal and professional responsibilities
39
3 Access, retrieve, evaluate, disseminate relevant information
44
4 Communicate and educate effectively
35
5 Manage drug distribution
80
6 Apply practice management knowledge and skills
50
Group Average (%)
59
56
47
53
74
63
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Validity – an ascent
from Practice Analysis to Test Results
Job/practice analysis
– Who/what contexts?
– How?
Test specifications & sampling
– Which competencies?
– Which tasks/scenarios?
– Other parameters?
Item writing and review
– Who and how?
Scoring
– Analytic (checklists) &/or holistic (scales)?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Validity – an ascent
from Practice Analysis to Test Results
Detect and minimize unwanted variability, e.g.:
– Items/tasks – does the mix matter?
– Practice effect – how can we avoid it?
– Presentation/administration – what is the impact of
different SPs, computers, materials/equipment?
– Scores – how do we know how accurate and dependable they are?
What can we do to improve accuracy?
Set Defensible Pass-fail Standards
– How should we do this when different standard setting methods > different standards?
– How do we know if the standard is appropriate?
Report Results
– Are they clear? Interpreted correctly?
– Are they defensible?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Validity – flying high
Evidence
– Strong links from job analysis to interpretation of
test results
– Relates to performance in training & other tests
Reliable, generalizable & dependable
– Scores
– Pass-fail standards & outcomes
Feasible
– Large & small scale programs
– Economic, human, physical, technological resources
Ongoing Research
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Wild Life
Candidate diversity
– Language
– Training
– Format familiarity,
e.g. computer skills
– Accommodations
Logistics
– Technological requirements
– Replications (fatigue, attention span)
Security
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
“Computer-Based
Simulations”
Karen S. Flint
Director, Internal Development &
Systems Integration
Applied Measurement Professionals, Inc.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Evolution of Simulation Exam
Format
• AMP’s parent company, NBRC, provided oral
exams from 1961 to 1978
• Alternative sought due to:
– Limited number of candidates that could be
tested each administration
– Cost to candidates who had to travel to
location
– Concern about potential oral examiner bias
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Evolution of Simulation Exam
Format
• Printed simulation exam format introduced in
1978 using latent image technology
• Latent image format used by NBRC from 1978
to 1999
• NBRC decision to convert all exams to
computer-based testing
• Proprietary software developed by AMP to
administer simulation exams in comparable
format via computer – introduced in 2000
• Both latent image test booklets & computerized
format being used
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
How Simulation Exams Differ from
MCQs
• Provides accurate assessment of higher order
thinking related to a content area of interest
(testing more than just recall)
• Challenge test takers beyond complexity of
MCQs
• Simulation problems allow test takers to assess
their skills against test content drawn from
realistic situations or clinical events
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Sample relationship between multiple-choice and
simulation scores assessing similar content
150
145
Passers
140
135
130
125
120
115
110
105
100
Failers
95
90
15
20
25
30
35
40
45
Scaled Simulation Score
Note: n = 68, r = .48, p <.001
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
50
Simulation Utility
• Continuing competency examinations
• Self-assessment/practice examinations
• High-stakes examinations
– Psychometric characteristics comparable to
other assessment methodologies
– That is, good reliability and validity
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Professions Using This Simulation
Format
•
•
•
•
•
Advanced-Level Respiratory Therapists
Advanced-Level Dietitians
Lighting Design Professionals
Orthotist/Prosthetist Professionals
Health System Case Management Professionals
(beginning 2005)
• Real Estate Professionals
• Candidate fees range from $200 to $525 for
full-length certification/licensure simulation
exam
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Structure of Simulations
• Opening Scenario
• Information Gathering (IG) Sections
• Decision Making (DM) Sections
– Single or multiple DM
• All choices are weighted (+3 to –3)
• Passing scores relate to judgment of
content experts on ‘minimal competence’
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Simulation Development
(Graphic depiction of path through a simulation problem)
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
IG Section Details
• IG section
– A section in which test takers choose information
that will best help them understand a presenting
problem or situation
– Facilitative options may receive scores of +3, +2, or
+1
– Uninformative, wasteful, unnecessarily invasive, or
potentially illegal options may receive scores of –1, –
2, or –3
– Test takers who select undesirable options
accumulate negative section points
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
IG Section Details
• IG Section Minimum Pass Level (MPL)
– Among all options with positive scores in a
section, some should be designated as
REQUIRED for minimally competent practice
– The sum of points for all REQUIRED options
in a section equals MPL
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
DM Section Details
• DM section
– A section of typically 4-6 options in which the test
taker must make a decision about how to handle the
presenting situation
– Facilitative options may receive scores of +3, +2, or +1
– Harmful or potentially illegal options may receive
scores of –1, –2, or –3
– Test takers who select undesirable options accumulate
negative section points and are directed to select
another option
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
DM Section Details
• DM Section Minimum Pass Level (MPL)
– May contain two correct choices, but one must
be designated as REQUIRED for minimally
competent practice
– The REQUIRED option point value in the
section equals MPL
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Minimum Passing Level
• DM MPL
– The sum of all DM section MPLs
• IG MPL
– The sum of all IG section MPLS
• Overall Simulation Problem MPL
– Candidates must achieve MPL in both
Information Gathering and Decision Making
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Simulation Exam Development
• 8 to 10 simulation problems per
examination
• Each problem assesses different situation
typically encountered on the job
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Let’s Attempt A Computerized
Simulation Problem!!!
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
• Karen S. Flint, Director, Internal
Development & Systems Integration
Applied Measurement Professionals, Inc.
8310 Nieman Road
Lenexa, KS 66214
913.541.0400
(Fax – 913.541.0156)
[email protected]
www.goAMP.com
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
“Practical Testing”
Dr. Frank Hideg, DC
Dr. Mark Christensen, PhD
Dr. Paul Townsend, DC
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
NBCE History
• The National Board of Chiropractic Examiners
was founded in 1963
• The first NBCE exams were administered in
1965
• Prior to 1965 chiropractors were required to
take chiropractic state boards and medical state
basic science boards for licensure
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
NBCE Battery of Pre-licensure
Examinations
•
•
•
•
Part I – Basic Sciences Examinations
Part II – Clinical Sciences Examinations
Part III – Written Clinical Competency
Part IV – Practical Examination for
Licensure
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Hierarchy of Clinical Skills
DO
PRACTICE
SHOW HOW
PART IV
KNOW HOW
PART III
KNOWLEDGE
PARTS I & II
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
NBCE Practical Examination
Content Areas
• Diagnostic Imaging
• Chiropractic Technique
• Chiropractic Case Management
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Content Weighing
TEC
17%
DIM
16%
DIM
CAM
TEC
CAM
67%
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Diagnostic Imaging
• 10 Four-minute Stations
• Candidate identifies radiological signs on
plain film x-rays
• Candidate determines most likely
diagnoses
• Candidate makes most appropriate initial
case management decisions
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Chiropractic Technique
• 5 five-minute stations
• Candidate demonstrates two adjusting
techniques per station
• Cervical spine
• Thoracic spine
• Lumbar spine
• Sacroiliac articulations
• Extremity articulations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Chiropractic Case
Management
• 10 five-minute patient encounter stations
• 10 linked post-encounter probe (PEP)
stations
• Candidate performs focused case histories
• Candidate performs focused physical
examinations
• Candidate evaluates patient clinical database
• Candidate makes differential diagnoses
• Candidate makes initial case management
decisions
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Key Features of NBCE
Practical Examination
• Use of standardized patients
• Use of OSCE format and protocols
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Case History Stations
• Successful candidates use organized
approach while obtaining case history
information
• Successful candidates communicate
effectively with patients
• Successful candidates respect patient
dignity
• Successful candidates elicit adequate
historical information
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Perform a Focused Case
History
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Post-Encounter Probe Station
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Part IV Candidate
Numbers
4500
4000
3500
3000
2500
2000
1500
1000
500
0
1996 1997 1998 1999 2000 2001 2002 2003
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Part IV State Acceptance
42
43
45
46
47
38
36
27
7
Colum n 1
1996
1997
1998
1999
2000
2001
2002
2003
2004
7
27
36
38
42
43
45
46
47
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
All shaded areas are those states that ACCEPT or REQUIRE successful
completion of the NBCE Part IV at the NBCE recommended passing
level of 375 as one portion of that state’s licensure evaluation process.
Those states are:
Alabama
Alaska
Arizona
Arkansas
California
Colorado
Connecticut
Delaware
Georgia
Hawaii
Idaho
Indiana
Iowa
Kansas
Kentucky
Louisiana
Maine
Maryland
Massachusetts
Minnesota
Mississippi
Missouri
Montana
Nebraska
Nevada
New Hampshire
New Jersey
New Mexico
New York
North Dakota
Ohio
Oklahoma
Oregon
Pennsylvania
Rhode Island
South Carolina
South Dakota
Tennessee
Texas
Utah
Vermont
Virginia
Washington
Wisconsin
Wyoming
* Those states requiring a passing score other than the NBCE
recommended score are not listed.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Candidate Qualifications
• Candidates must pass all basic science and
clinical science examinations before
applying
• Candidates must be within 6 months of
graduation from an accredited
chiropractic college
• $1,075 examination fee
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Contact Information
• National Board of Chiropractic Examiners
901 54th Avenue
Greeley, CO 80634
970-356-9100, 970-356-1095
[email protected]
www.nbce.org
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Station
Development
Alison Cooper
Manager of Examination Operations
Canadian Alliance of Physiotherapy
Regulators
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
First Principles
• If it’s worth testing, it’s worth testing well
– it is possible to test anything badly
– this is more expensive
• Some things are not worth testing
– trivia
– infrequently used skills
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Overview
•
•
•
•
Write
Review
Dry run
Approve
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Write
•
•
•
•
•
•
•
Focus of station
SP portrayal - general
Checklist & scoring
Instructions to candidate
Details of SP instructions
Review everything
References
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Focus of Station
• Each station must have a clear focus
– establish the focus in one sentence
– take time to get this right
– you can’t write a good station without a clear
focus
• Example: Perform passive range of motion
of the arm for a client who has had a
stroke.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
SP Portrayal - General
• Consider SP movement, behaviour
– a picture in your head
– use real situations to guide you
• Not detailed yet
• Example: Client is 55 years old, is
disoriented, and has no movement in the
left arm or leg.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Checklist & Scoring
•
•
•
•
•
What is important to capture
Consider the level of the candidates
Group items logically
Assign scores to items
Scoring scales
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Checklist Example
•
•
•
•
•
•
•
•
•
•
Explains purpose of interaction
Corrects client’s position
Performs passive ROM of scapula
Performs passive ROM of shoulder
Performs passive ROM of elbow
Performs passive ROM of wrist
Performs passive ROM of hand & fingers
Performs passive ROM of thumb
Uses proper body mechanics
Uses proper handling
1
2
1
1
1
1
1
1
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
3
3
Instructions to Candidate
• Information the candidate needs
– age and sex of client
– pertinent information and assumptions
• The task for the candidate
– exactly what they are to do and not do
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Example
Eric Martin
55 years old
This client had a right middle cerebral artery haemorrhage
resulting in a left sided hemiplegia two (2) weeks ago.
The client presents with confusion and left sided flaccidity. His
cardiovascular status is stable.
• Perform passive range of motion on the client’s left
upper extremity.
• Perform only one (1) repetition of each movement.
• Assume that you have the client’s consent.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Details of SP Instructions
• History, onset, changes
• Initial position, movements, demeanor,
must say/ask
– anticipate strong AND weak candidates
• Cover the checklist and candidate
instructions
• SP prompts
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
SP Instructions...
• Use plain language
• Include
– what to wear/not wear
– features of the SP (height, scars)
• Diagrams are often helpful
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Example
•
•
•
•
•
•
•
•
•
•
•
Presenting complaint
Initial position, general mobility, affect
Comments you must make
Medical, social history
Medications
Activities and areas affected
Sensation
Pain
Muscle quality
Responses to candidate
Emotions
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Check Everything
• Go back and check
– does it make sense?
– is there still a clear focus?
– is anything missing?
• Edit/revise as needed
– add notes to examiner for clarification
• Check for plain language
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
References
• Use references you expect candidates to
know
• Umphred, 2nd edition, page 681
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Next Steps
• Review by others
• Dry run
• Approve for use
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Thank you
Canadian Alliance of Physiotherapy Regulators
1243 Islington Ave., Suite 501
Toronto, ON, Canada M8X 1Y9
(W)416-234-8800, (F)416-234-8820
[email protected]
www.alliancept.org
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
“OSCE Research: The Key to
a Successful Implementation
”
Lila J Quero Muñoz, PhD
Consultant
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Prior to the OSCE:
CPBC and PEBC
• Need for assessing communication,
counseling, and interpersonal skills to
provide pharmaceutical care to patients
• PEBC MC examination was not assessing
the full scope of pharmacy practice as
profiled by NAPRA (National Association
Pharmacy Regulatory Authorities of
Canada)
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Generalizability: Data Analyses
• Psychometrically, OSCEs, are complex
phenomena, producing scores with
potential errors from multiple sources,
including:
–
–
–
–
–
Examiners (pharmacists and non-pharmacists)
Cases (context, complexity, # of stations)
Scoring methods (global vs. checklists)
Standard setting
Differential grading practices
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Research Question # 1
• How many examiners are required to
obtain consistent and dependable
candidates’ scores?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results #1-1998
• 1 examiner per case yielded similar
consistency as 2 (G=.82, .81, D=.81, .79)
indicating that examiners agreed highly on
their scores
• Examiners contributed little to the scoring
errors of candidates’ performance
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
1 vs. 2 Global -1999
Global Grading Roving & Assessor G and D Reliability as a
Function of Stations & Raters
Mean of 3 Sets of Stations
G & D Reliability Estimates
G Rel 1Rater
G Rel 2 Raters
1
D Rel 1 Rater
0.9
D Rel 2 Raters
0.8
0.7
0.6
0.5
0.4
0.3
3
8
10
13
15
17
20
# of Stations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Research Question # 2
• How many cases are required to maintain
consistency, validity and generalizability of
scores?
– Adequate and representative sampling of
professional practice are necessary to capture a
candidate’s abilities.
– Multiple observations of abilities yield more
consistent and content valid inferences.
– Logistical constraints restrict the number of cases
that are timely and economically feasible to
administer within one OSCE examination.
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 2-1998
• 15 cases reduced the candidate’s score
error due to sampling variability of the
cases dramatically from 5 or 10 cases and
improved the consistency of scores from
G=.60 to .81
• 15 cases reduced the cases and raters
interaction variance as an indication that
raters agreed on their scores across cases
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 2-1998
• Candidates’ scores varied mostly due to
their differential performance across cases.
• Sampling of the cases might affect the
candidates’ performance on an OSCE.
• We suggest, however, that differential
performance across cases might be due to
candidate’s differential levels of skills across
the pharmacy competencies assessed
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Profile of Sources of
Errors in %-1998
45
True score variance
40
35
30
25
20
15
10
5
0
Rater variance
Case variance
Interaction can. &
rater
Interaction can. &
case
Interaction rater &
case
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Research Question # 3
• How do different scoring methods such as
checklists or global grading affect
candidates’ scores?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 3-1998
• Low correlations between checklist and
global scores suggest both methods might
not be interchangeable
• If used in isolation they would yield
different end results, particularly for
borderline candidates
• Global grading yields higher mean scores
than checklist grading (p values.81 and
.59)
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Global vs. Checklist-1999
C o mp a r i n g H o l i s t i c ( G l o b a l ) & C h e c k l i s t G r a d i n g G a n d D
G &D
Estimates
R e l i a b i l i t y ( A s s e s s o r & St a n d a r d i z e d P a t i e n t ) / O n e R a t e r
0.9
0.7
0.5
0.3
8 10 12 14 16 18 20
Stations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
G1 Check
Nest Rel.
D1 Check
Nest Rel.
G1 Global
Nest Rel.
D1 Global
Nest Rel.
Research Question # 4
• What is the validity and defensibility of
standard-setting procedures and pass/fail
decisions
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 4-1998
• SME’s agreed highly on the minimum standard
necessary for safe pharmacy practice for the
borderline qualified pharmacists
• On different occasions, SME’s had similar
standards for entry-to-practice for the core
cases
• Standards varied little between 26 & 20 cases
and were consistent enough with 15 cases
(G=.74, .74, .71)
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 4-2003
Standard Setters (SS) Variation Within Station and Mean
4.00
ss1
3.00
ss2
2.50
ss3
2.00
ss4
1.50
ss5
1.00
ss6
ss7
0.50
ss8
0.00
ss9
J0
10
J0
72
G0
08
J0
58
Ri
30
B
Ro
09
J0
73
J0
68
J0
44
J0
42
J0
37
Q0
12
J0
65
P0
06
W
00
1
M
ea
n
SS Mean Scores
3.50
Stations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
ss10
MEAN
Research Question # 5
• Are there differential grading practices
among Canadian Provinces?
• Are candidates’ pass/fail decisions affected
by provincial differences on scoring
practices?
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 5-Videos 2003
• Variability in scores between sites are due
mostly to true score variance
• Differences between exam sites are in
magnitude of scores but not in pass/fail status
• Differences between assessors are mostly of
magnitude of scores but not in pass/fail status
• Pass/Fails decisions did not vary between sites
and assessors
• There is more variance between assessors than
between exam sites
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 5-2003
Stations J03 & J29 Spring 03
Assessors' Mean
Scores
4
3.5
3
J03
2.5
J29
2
1.5
1
101
103
401
402
701
OSCE Sites and Assessors
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Results # 5-2003
Global Grading Spring Exam Sites 10,
40
Global Grading Spring Exam Sites 10,
40
J03
J29
Facets
df
var
SE
P
18
0.150
0.077
L
1
0.058
R:L
2
PL
PR:L
Total
Facets df
var
SE
25%
P
14
0.267
0.104
69%
0.069
10%
L
1
0.000
0.015
0%
0.029
0.035
5%
R:L
2
0.038
0.030
10%
18
0.000
0.066
0%
PL
14
0.026
0.021
7%
36
0.368
0.085
61%
PR:L
28
0.057
0.015
15%
100%
Total
0.606
%var
0.388
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
%var
100%
Results # 5-2003
Assessors' Mean Scores
124 Videos (6 Stations Spring 03 Exam)
Scored by 35 Raters from 9 Exam Sites
(Passing Score 2.786)
4
3.8
3.6
3.4
3.2
3
2.8
2.6
2.4
2.2
2
1.8
1.6
1.4
1.2
1
MJ33
MRa02
MJ38
MJ45
MJ03
Stations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
MJ29
Conclusions 1998-2004
• Development of cases should follow
templates, guidelines and a detailed
blueprint
• Selection of cases must follow a detailed
blueprint to mirror OSCE forms between
exam administrations to control for
differences in cases such as complexity
and content
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Conclusions 1998-2004
• Multiple sources of errors in OSCEs
forces us to do more extensive and nontraditional research than for MC exams
• OSCEs require continuous vigilance to
assess the impacts of the many sources of
errors
• OSCE research must be planned and
implemented beyond exam
administrations
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Conclusions 1998-2004
• OSCE infrastructure must support both
design research and exam administration
research
• Successful implementation and continuous
improvements of OSCE go hand and hand
with research
• More collaborative efforts among OSCE
users are needed to built on each other’s
success and avoid pitfalls
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
Conclusions 1998-2004
• Although OSCE research is costly it is a
deterrent to litigation and wasted exam
administration resources
• Similar conclusions may apply to other
performance assessments
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri
• Carol O’Byrne, BSP, OSCE Manager
• John Pugsley, PharmD, Registrar, PEBC
[email protected]
416-979-2431, 1-416-260-5013 Fax
• Lila J. Quero-Muñoz, PhD, Consultant
787-431-9288, 1-888-663-6796 Fax
[email protected]
Presented at the 2004 CLEAR Annual Conference
September 30 – October 2
Kansas City, Missouri