Using the Mini-Clinical Evaluation Exercise (Mini-CEX) as an assessment tool for medical students April 29, 2011

Why consider the Mini-CEX?

Year 4 OSCE 2011 Overall Results • Number sitting OSCE • Number passing 248 245 (98.8%)

Competency Domains Reliably Measured Competencies/Skills Counselling History-taking Other Competencies and Skills Measured Physical examination Diagnosis Management Lab data Communications Professionalism

Class Mean SD Requires Improvement Alpha

70 65 10 5 35 38

.63

.72

78 71 66 80 77 77 8 13 8 11 3 3 3 46 53 17 0 0

.75

.15

.02

.21

.69

.48

What is mini-CEX?

• Structured 10 min observation of a student performing specified tasks during routine practice • Feedback session (10 min) • Completion of a standardized one-page rating form

Mini-CEX: Types of tasks

• Focused history • Physical examination • Counselling – advise patient regarding management options – provide appropriate education – make recommendations that address patient’s concerns • Clinical reasoning skills – diagnostic and therapeutic skills • Case presentation

History-taking Form

Guidelines for marking

1 0.8

0.6

0.4

0.2

0 1 Reliability of the Mini-CEX Average Composite Ratings

Changes in reliability as a function of the observed number of encounters

4 8 12 16 20

Number of encounters

24 28 32 36

Implementing a Mini-CEX Assessment Process

Implementing a Mini-CEX

• Orientation – Familiarize yourself with the mini-CEX rating form and definition of the components of the student’s performance you will be rating • Schedule the mini-CEX – 1-2 / week – Allow 20 minutes for each assessment – Obtain patient permission

Implementing a Mini-CEX

• Select the patient encounters to be observed – Year 3 “must see” list of medical conditions • new or existing medical problem • acute vs. chronic illness • different age groups and both genders • different clinical settings (e.g. office, hospital) if possible – Ask the student to perform the task without prompting about the possible diagnosis • perform an abdominal examination • Not: examine the patient for possible appendicitis

Assessment Process

• Avoid interrupting the student during the patient encounter – no questions, comments or suggestions – if you want to follow-up findings with patients, do this after the student is finished • Conduct immediate feedback (10 minutes) • Complete rating form • Discuss rating or comments with student

Mini-CEX: Feedback

• Immediate • Specific • Limited to key issues • Honest • Fair • Descriptive, not judgmental, e.g.

“you did not examine

X

”

NOT “

you were way off base”

Mini-CEX: Feedback

• Two-way process (inter-active) • Start by asking the student some questions –

how they felt they did with the patient

–

what findings they found

–

what they think is the most likely diagnosis

–

why they ordered a particular investigation or suggested a particular treatment

• Answers can stimulate specific feedback and also guide ratings of performance

Feedback challenge

• Easy when the student does well • More difficult when the performance is poor – do not hesitate to point out area of weakness – multiple assessments with multiple examiners (reliability) – sampling performance across the spectrum of clinical situations (validity)

Mini-CEX: Feedback

Closing the loop Provide a recommendation – interviewing/ examination/ counselling skills/ management/ presentation • Develop a specific action plan – allows student to act on the recommendation

Example of use of Mini-CEX

ER • A man presents with abdominal pain • The student performs a focused abdominal examination (10 minutes) • The preceptor notices that the student did not examine the inguinal areas adequately • Feedback is given – the preceptor demonstrates the correct technique – recommends a review of hernias in clinical skills textbook – suggests plan to practise exam technique

Summary: Key features of the Mini-CEX • Direct assessment of actual patient care • Allows assessment of performance • good evidence supporting mini-CEX’s validity and reliability • cumulatively can infer student’s competence • Can be incorporated into daily activities • efficient use of resources • Allows immediate and substantive feedback

Assessor’s training

• Paper-based orientation – Familiarize with the process, specific observation task and ratings form – All assessors who participated received this form of training • Workshop* - video-based training – Videos exemplifying three levels of performance – Rated at the end on the form by all participants *(Modeled after ABIM/NBME ‘Direct observation of Competence Training Program’, Holmboe et al., 2004)

Effects of training

• Assessors of the post graduate trainees – Raters not trained in workshop were more lenient: 3.17 vs. 2.31 6.17 vs. 4.85 8.29 vs. 7.38

– Both the scenario and training-group effects were significant

Reliability

• Generalizability-theory approach to reliability – Allows for estimating the variance-components attributable to the different factors of the measurement situation – Calculating G-coefficient (reliability coefficient) – Modeling the effect of changes in these factors (e.g., number of items needed to achieve certain level of precision) • Number of mini-CEXs needed was the main factor followed through all studies (in one case, the effect of blueprinting was also explored)