Transcript St George’s Advanced Assessment Course thoughts on a final medical assessment – the Short Case GAME 5th December 2012 Yvonne Finn.

St George’s
Advanced Assessment Course
thoughts on a final medical
assessment – the Short Case
GAME 5th December 2012
Yvonne Finn
Briefly...........
•
•
•
•
•
The Short Cases examination
Validity and quality assurance
Constructs, domains & descriptors
Standard setting
Pros and cons of whole process
5MB Short case
•
•
•
•
•
•
•
30% of 5MB year assessments
Requirement to pass
OSCE like in set-up
6 stations; 1 pair of examiners at each
No set script
Usually clinical skills, diagnosis, interpretation
A single GRS
Short case marking sheet
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
YEAR 5 FINAL CLINICAL EXAMINATIONS 2011/2012
SHORT CASES
STUDENT NAME:__________________________________________
STUDENT ID NUMBER:_____________________________________
DAY:
SESSION:
1 - WEDNESDAY
MORNING
2 - THURSDAY
AFTERNOON
(PLEASE CIRCLE)
(PLEASE CIRCLE)
STATION NUMBER (1 – 6): _________________
AGREED GLOBAL MARK (OUT OF 10, to the nearest 0.5)*:_________
COMMENTS (if any):
EXAMINERS’ SIGNATURES: _______________________
_____________________
*Please consider the following elements of the student’s performance in deciding upon his/her mark for this station: communication skills, approach to physical
examination, ability to elicit abnormal physical signs, ability to interpret physical findings. Please use the full range of marks from 0 to 10 (to the nearest 0.5),
referring to the following broad grade brackets as a guide: 0-4 = CLEAR FAIL; 4.5 = BORDERLINE PASS/FAIL; 5/5.5 = PASS; 6/6.5 = SECOND CLASS HONOURS; 7-10 =
FIRST CLASS HONOURS. The gold medal viva candidate is likely to score well in excess of 7/10 in most stations.
Validity
1. “Validity is the sine qua non of assessment , as
without evidence of validity , assessments in
medical education have little or no intrinsic
meaning”(1)
2. Contemporary view of validity: a unitary
concept, with construct validity as the whole of
validity
3. Purpose of validity testing is to support (or
refute) the proposed interpretations of the test
scores.
1. Downing S Med Ed 2003; 37: 830-837
Validity in Clinical Assessment
• “Validity depends on the fidelity of the
simulation”
• Validity depends on the implementation of the
assessment”
Van der Vleuten, St Georges;
Advanced Assessment Course
Validity testing by...
• Map test content to learning objectives of
modules
• Content representativeness of test material
• Peer review
• Examination blueprinting (handout)
• Accurate descriptions/interpretations (ET)
• Internal structure (reliability e.g. GT)
Validity testing by...
• Response process – documentation of data
quality control; rationale for using GRS vs
checklist
• Relationship to other variables (correlations)
• Consequences – impact for examinees for
assessment scores and outcomes (FP & FN)
e.g. determination of the pass score,
examiner qualifications, number of examiners
Constructs
• “intangible collections of abstract concepts
and principles which are inferred from
behaviour and explained by educational or
psychological theory”
• e.g. Educational achievement
Educational ability (aptitude)
Attitudes
Domains in performance-based
assessments
Examples (handout)
Clinical content – information gathering (Hxtaking)
Clinical content – giving information
Clinical content – communication skills
Physical examination – clinical skills
Procedure – clinical skills
GRS
• Checklist vs GRS
• GRS – “subjective expert judgement”
more reliable than checklists
“use expert judgement for assessing
complex skills”
• Use of domains (handout with descriptors)
Standard setting
• Purpose - identify the cut-score, minimum
competence required for progression/
graduation
Competent Incompetent
Pass
Fail
Standard Setting
• Method has to be reliable, credible, supported
by a body of evidence in the literature,
feasible, acceptable to all stakeholders
• Absolute standard/criterion referenced
• Borderline regression method with GRS
requires at least 4 domains of GRS + overall
GRS e.g. Handout; at least 100 students
Thoughts on the process
•
•
•
•
•
CONSENSUS
DOCUMENTATION
VEHICLE / STRUCTURE
INCREMENTAL
RESOURCES