Transcript OSCEs - IDEAL Consortium

OSCE
Kathy Boursicot
Train the Trainer Assessment Workshop
October 29, 2003
Hong Kong International Consortium
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
What is an OSCE?

Objective

Structured

Clinical

Examination
Harden RG and Gleeson FA Assessment of clinical competence using an
objective structured clinical examination (OSCE) Medical Education,1979,
Vol 13: 41-54
OSCE test design
Observed Stations:
clinician examiners
Varieties of OSCEs
Patient-based
Traditional OSCE
SP-based test
Station couplets
Integral consultations
Clinical task
Written task
….
….
….
….
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
Professional authenticity
Simple model of competence
Does
Shows how
Knows how
Knows
Miller GE. The assessment of clinical
skills/competence/performance. Academic
Medicine (Supplement) 1990; 65: S63-S7.
Testing formats
Behaviour~
attitude/skills
Professional practice
Does
Shows how
Cognition~
knowledge
Knows how
Knows
OSCEs
EMQs, SEQs
MCQs
OSCE - Objective

All the candidates are presented with the
same test

Specific skill modalities are tested at each
station

History taking

Explanation

Clinical examination

Procedures
OSCE - Structured

The marking scheme for each station is
structured

Structured interaction between examiner
and student
OSCE – Clinical Examination

Test of performance of clinical skills

candidates have to demonstrate their skills,
not just describe the theory
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
Characteristics of assessment instruments

Utility =
 Reliability

Validity

Educational impact

Acceptability

Feasibility
Van der Vleuten, C. The assessment of professional competence:
developments, research and practical implications,
Advances in Health Science Education, 1996, Vol 1: 41-67
Test characteristics

Reliability of a test / measure
 reproducibility
of scores across
raters, questions, cases, occasions
 capability
to differentiate consistently
between good & poor students
Sampling
Domain of Interest
Test Sample
Test Sample


Reliability

Competencies are highly domain-specific

Broad sampling is required to obtain
adequate reliability

across content, i.e., range of cases/situations

across other potential factors that cause error
variance, i.e.,

testing time, number of cases, examiners, patients,
settings, facilities
Test characteristics

Validity of a test / measure

The content is deemed appropriate by the
relevant experts

The test measures the characteristic (e.g.
knowledge, skills) that it is intended to
measure

The performance of a particular task predicts
future performance
Test characteristics

Validity of a test / measure
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
Advantages of using OSCEs in clinical assessment

Careful specification of content = Validity

Observation of wider sample of activities =
Reliability

Structured interaction between examiner & student

Structured marking schedule

Each student has to perform the same tasks =
Acceptability
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
OSCE Station Writing
How to start

Decide what tasks you



want to
can
should
test in an OSCE format

OSCEs test performance, not knowledge
Constructive alignment

Need to know the learning objectives of
your course / programme

Map these across :

Subject areas

Knowledge areas

Skill areas
Blueprinting

Content of the assessment should align
with the learning objectives of the course

Blueprinting

allows mapping of test items to specific
learning outcomes

ensures adequate sampling across subject
area and skill domains
OSCE blueprint: systems-based
Hx taking
(incl. diag)
CVS
Endocrine
Gastro
H&N
Haem & LN
Musculoskl
etc
Phys exam
(incl. diag)
Procedures
Counseling/E
ducation
Ordering
investigs
OSCE blueprint: discipline-based
Hx taking
(incl. diag)
Anaes & CC
Clin Pharm
Comm Health
Emergency med
Family med
Musculoskel
etc
Phys exam
(incl. diag)
Procedures
Counseling/E
ducation
Ordering
investigs
Key features of success in designing OSCEs

Feasibility

Congruence
Feasibility

Is it a reasonable task to expect the
candidates to perform?

Can the task be examined at an OSCE
station?

Can the task be performed in the time
allowed?
Feasibility

Is it a reasonable task to expect the candidates to
perform? Is it authentic?

Can the task be examined at an OSCE station?
 Match clinical situations as closely as possible
 Some tasks may require simulated patients
 Some tasks may require manikins
 Some tasks simply cannot be examined in this format
Feasibility

Can task be performed in time allowed?

Pilot the stations to see if they are feasible

Check equipment /helpers/practicalities
Congruence

Is it testing what you want it to test?

Station construct: describe what station
is testing
Congruence

Ensure that all parts of station coordinate

Candidate instructions

Marking schedule

Examiner instructions

Simulated patient instructions

Equipment
Station construct

This station tests the candidates ability
to …………………………
Candidate instructions

State circumstances: e.g. outpatient clinic, ward,
A & E, GP surgery

Specify the task required of the candidate: e.g.
take a history, perform a neurological
examination of the legs, explain a diagnosis

Specify tasks NOT required

Instruct on summing up: e.g. tell the patient, tell
the examiner
Examiner instructions

Copy of candidate instructions

Specific instructions appropriate to the task:

e.g., do not prompt, explicit prompts, managing
equipment
Simulated patient instructions

Give as much detail as possible so they can be
consistent

try to leave as little as possible for them to ad lib!

Give enough information to enable them to
answer questions consistently

Be specific about affect in each role

Specify patient demographics

i.e., gender, age, ethnicity, social class, etc.
Marking schedule

Ensure marks are allocated for tasks the
candidates are asked to perform

Decide relative importance of diagnosis vs
process (history taking, examination)

Separate checklist for process skills
Equipment


Be detailed
Think of



Chairs + table / couch / bench
Manikins - specify
Medical equipment

Stethoscope, ophthalmoscope, sphyg,
suturing materials, etc
Designing stations

Use your blueprint

Be clear what you are testing: define the
construct

Check for congruence

Pilot for feasibility
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
Training observers

Understand the principles of OSCEs

Enhance inter-rater consistency
Techniques

Examiners must train together



Videos
‘live’ stations
Discussion of marking inconsistencies
Training observers

General training

Station-specific training
OSCE

Format

Purpose

Advantages

Writing principles

Training observers

Scoring considerations
Scoring considerations

Global vs checklist scoring

Weighting

Standard setting
Checklist scoring


Advantages

Helps examiner know what the station setters are
looking for

Helps the examiner be objective

Facilities the use of non-expert examiners
Disadvantages

Can just reward process/thoroughness

May not sufficiently reward the excellent candidate

Ignores the examiners expertise
Global scoring


Advantages

Utilises the expertise of the examiners

They are in a position to make a (global) judgement
about the performance
Disadvantages


Examiners have to be expert examiners i.e. trained
Examiners must be familiar with expected standards
for the level of the test
Weighting

In a checklist, some items may be weighted
more than others

More complicated scoring system

Makes no difference to very good & very bad
candidates

Can enhance discrimination at the cut score
Standard setting

No perfect method!

Should be criterion-referenced method

e.g. Angoff, Ebel, etc.
But
 are these suitable for performance based tests?
Performance-based standard setting methods

Borderline group method

Contrasting group method

Regression based standard method
Kramer A, Muijtjens A, Jansen K, Düsman H, Tan L, van der Vleuten C
Comparison of a rational and an empirical standard setting procedure for an OSCE,
Medical Education, 2003 Vol 37 Issue 2, Page 132
Borderline method
Test score distribution
Checklist
1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld


4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as
TOTAL



Borderline score distribution
Pass, Fail, Borderline P/B/F
Passing score
Contrasting groups method
Test score distribution
Checklist
1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld


4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as
TOTAL


Fail
Pass

Pass, Fail, Borderline P/B/F
Passing score
Regression based standard
Checklist
X = passing score
1. Hs shjs sjnhss sjhs sjs sj
2. Ksks sksmsiqopql qlqmq q q qkl
3. Lalka kdm ddkk dlkl dlld


Checklist
Score
4. Keyw dd e r rrmt tmk
5. Jfjfk dd
6. Hskl;s skj sls ska ak akl ald
7. Hdhhddh shs ahhakk as


X

TOTAL
Overall rating 1 2 3 4 5
1
2
3
4
5
=
=
=
=
=
Clear fail
Borderline
Clear pass
v good pass
excellent pass
1
2
3
Clear
fail
Borderline
Clear
pass
4
5
v good pass excellent pass
Borderline/contrasting/regression based methods

“Panel” equals examiners

Reliable: cut-off score based on large sample of
judgments (no. of stations x no. of candidates)

Credible: based on expert judgment in direct
observation

Passing score not known in advance
(as with all
examinee centered methods)

Judgments not independent of checklist scoring