Transcript Introduction to Evaluation

The MiniCEX and the
Evaluation of Clinical Skills
National Health Service
Foundation Training Program
Patient Care Competencies

Gather accurate, essential information from
all sources, including medical interviews,
physical examinations, medical records and
diagnostic/therapeutic procedures
 Make informed recommendations about
preventive, diagnostic and therapeutic
options and interventions that are based on
clinical judgment, scientific evidence, and
patient preference.
Patient Care: Themes

Clinical skills essential to patient care
 Cannot make “good” decisions unless you
work with good and accurate information
– GIGO principle

Evaluation of clinical skills requires direct
observation
Workshop Objectives

Review current state of
– Physician clinical skills
– Faculty evaluation skills
 Understand the importance of direct
observation by faculty for assessment
of clinical skills
Workshop Objectives

Discuss practical strategies for focused
direct observation
 Review rater training methods
– Direct Observation of Competence
(DOC) training
Workshop Elements

Mini-Lectures:
– State of clinical skills
– Quality of faculty ratings

Direct observation exercises
– Performance dimension exercise
– Frame of reference training
Video Exercise

Situation: A foundation trainee performs
a medical interview in the outpatient
setting. Using the MiniCEX form
provided, please rate the performance of
this trainee.
Key “Basic” Clinical Skills

Medical interviewing
 Physical examinations
 Counseling/patient education
 Clinical judgment/reasoning
 Reflective practice
– Self-directed learning
– Professional growth and improvement
– Medical errors
Are Clinical Skills Important?

Where do clinical skills fall into the hierarchy
of physician competencies and mastery in an
era of advanced technology?
Diagnosis and Medical Interview

Hampton (BMJ, 1975):
– Medical interview: 82%
– Physical exam: 9%
– Laboratory: 9%

Kirch (Medicine, 1996)
– Medical interview (+PE): 70%
– Imaging: 35%
Importance of Sound Clinical Skills

Diagnostic errors
– Inaccurate/ incomplete medical interview
one of leading causes (Bordage)
• Wrong information leads to wrong decisions

Patient satisfaction
– Higher with better communication skills

Patient self-care
– Better adherence and outcomes associated with
better physician communication skills
Clinical Reasoning: A Primer
Patient/situation
characteristics
Prior knowledge
Problem Representation*
Evaluation
Action
Information Gathering
Context
Gruppen and Frohna, International Handbook on Research, 2002
Clinical Skills: U.K. Trainees

Fox (2000)
–
–
–

Voluntary study of 22 PRHOs using OSCE
Only 45% with passing score on drug advice
communication station
0% passed locomotor system examination
Evans (2004)
– 26 new PHROs
– All had passed 22 station OSCE in medical school
– Majority failed skill stations in blood pressure
measurement and cannulation
Clinical Skills: U.S. Trainees

Stillman (1990)
– OSCE: wide variability in graduating
medical student clinical skills

Mangione (1997)
– Deficient cardiac and pulmonary
auscultatory skills
– Medical students, FP and IM residents
– Replicated findings in Canada and U.K.
Clinical Skills: Practicing MDs

Ramsey (1998)
– Incomplete history-taking / preventive health
screening among Primary care physicians in
Northwest Unites States

Braddock (1999)
Study of informed decision making (IDM) and
counseling
– Simple analysis of presence or absence of 7 key
elements
– 1058 outpatient visits: only 9% of visits met minimal
criteria for IDM
–
Importance of Faculty: U.K. Studies

Grant (Med Educ, 2003)
–
–

Inadequate coverage and frequency of
supervision activities
Discordance between specialist registrars and
attendings
Kilminster (Med Educ, 2000)
–
–
Systematic review of supervision
Better supervision associated with improved
patient safety and quality of care.
Importance of Faculty: U.S. Studies

Inpatient Study (Lancet, 2003)
–
–

Reviewed 100 consecutive admissions to
teaching service in U.S.
Faculty detected 26 PExam findings missed by
house officer that impacted patient’s care
Outpatient Studies
–
Two separate studies showed that faculty
assessment disagreed with that of house
officer in up to 30% of patients
Clinical Skills: Themes

Deficiencies exist across continuum
 Specific skills more “error-prone”
– Eg: musculoskeletal and neuro exams

Not detected by other evaluation methods
– Performance of basic clinic skills does
not correlate with performance in other
dimensions of competence
Clinical Skills: Themes

House officers:
– Aware of importance
– Recognize under-emphasis
 Without detection deficiencies in
clinical skills cannot be corrected
Miller’s Pyramid
Portfolios
DOES
SHOWS
HOW
KNOWS HOW
KNOWS
Faculty Observation
OSCE
Extended matching /
CRQ
MCQ EXAM
Faculty Observation / Rating Skills

Patient care settings
– Ratings based mostly on perceived
knowledge and personality
– Little evidence of direct observation
– Significant “Halo” effect
Gray, Thompson, Haber, Grant, etc.
Faculty Observation / Rating Skills

Research settings
– Poor inter-rater reliability
– Brief rater training methods ineffective
• Didactic instructions
• Demonstration videos without practice
– Accuracy: structured > open-ended forms
– Increased accuracy  discriminative ability
Kalet, Herbers, Noel, Kroboth
Faculty as Raters – Key Issues


Faculty do not observe actual
performance
Faculty ratings lack:
– Reliability
– Accuracy/validity

Content specificity
–
How comfortable are you with own skills?
Improving Faculty Ratings: Solutions

Step 1: Getting faculty to observe
– Required a part of Foundation Program
– Focused observations are logistically possible
– 5 to 10 minute observations are valuable
– Build on faculty “epiphany”
• The “you will not believe what I saw
today” experience
– Provide “usable tool”
Foundation Mini - CEX Tool

Simple rating scale using 6 dimensions and
overall rating
 “Structured” approach to direct observation
 Direct assessment of actual patient care
 Incorporation of CEX into daily activities
 Evidenced-based
Research: Mini - CEX Tool





Two large scale U.S. studies involving 36 total
residency programs
Logistically feasible to incorporate miniCEX
into daily activities
High satisfaction among house staff
Good to excellent reliability characteristics
Overall scores and interpersonal scores
correlated with trainee’s ECFMG OSCE scores
Logistics: Outpatient Clinic

One mini-CEX per trainee per day per
week
– One attending observes portion of first visit
of the day
– Minimizes disruption of clinic
– Perform over course of academic year
– Easy to obtain 6-8 Mini-CEX’s per year per
trainee in single setting
The Patient Encounter

Sampling “parts” of the encounter:
INTERVIEW
PHYSICAL
EXAM
COUNSELING
Solutions: Step 2

Improve reliability
– Multiple brief observations
– Perform over time: outpatient setting allows
for longitudinal observation
– Involve multiple faculty
– MiniCEX: sufficient reliability for pass/fail
determinations after just 4 observations
Solutions: Step 3

Improve accuracy and validity
– Most difficult step
– Use structured rating forms
– Rater training (faculty development)
• Caveat: brief “one time”
interventions do not work
Does Faculty Training Work?
Performance Appraisal Literature:
 Can reduce rating errors
 Can improve discriminative ability
 Can improve accuracy
Approaches to Rater Training

Behavioral Observation Training
 Performance Dimension Training
 Frame of Reference Training
 Direct Observation of Competence
Training
Videotape Exercise: BOT

Situation: An attending is performing a
miniCEX of a house officer performing a
physical exam.
 Questions:
– How well did this attending evaluate the
house officer?
– How was the house officer-patient
interaction affected?
Basic Faculty Observation Skills

Prepare for the observation
– Faculty: Know what you’re looking for
– Resident: Let them know what to expect
– Patient: Let them know why you are there
Minimize intrusiveness – correct positioning
 Minimize interference with the house officerpatient interaction
 Avoid distractions

Triangulation
DESK
Patient
Resident
Attending
Basic Observation Strategies
 Increase the amount of “sampling”
– More observations lead to more accurate
evaluations (“practice makes perfect”)
 Use of observational “aides”
– Behavioral diary to record observed
performance.
– U.S. study: simple 3X5 card diary lead to
increased comments on forms
Performance Dimension Training

Group exercises designed to familiarize
faculty with the specific elements of a
competency
 Should involve discussion of the criteria
required for each element
 Use defined, agreed upon elements of a
competency to calibrate faculty
– Playing from the “same sheet of music”
PDT Exercise
In your small group, discuss what should
be the basic components of an effective
medical interview for a foundation
trainee performing an outpatient
consultation
Frame of Reference Training
Goal is to improve “judgment” and accuracy
Steps in FOR training:
1. Group performance dimension training (PDT)
exercise
2. Review clinical vignettes that describe critical
incidents of performance: unsatisfactory to
average to superior

Frame of Reference Training
3. Faculty, using framework developed in PDT
exercise, provide ratings on a behaviorally
anchored rating scale (BARS)
4. Session trainer provides feedback on what
“true” ratings should be for each vignette
along with rationale
5. Group finishes by discussing discrepancies
between trainer’s ratings and the
participants’ ratings
Frame of Reference Training

Most difficult aspect of FOR:
– Setting the actual standards that
distinguish between levels of
performance
– Reaching agreement and/or consensus
among teaching faculty
DOC Training

Combination of:
–
–
–
–
Behavioral observation training
Performance dimension training
Frame of reference training
“Live” practice in observation with
standardized residents/patients
• Individual evaluation and feedback
• Group debrief with Eval and FB
DOC Training Trial

Randomized controlled trial of 40 faculty
from 16 residency programs
 DOC training:
–
–
–
–
High satisfaction (favorite aspect of course)
Increased comfort in observation
Changed rating behavior at 8 months
Increased accuracy in identifying
unsatisfactory performance
Direct Observation: Challenges

Like all skills, requires training and
practice
 Faculty “calibration” important
– Agreeing on “metrics” of performance
– Faculty comfort with own skills

Faculty training
– Brief interventions mostly ineffective
Observation: Helpful Hints

Sample “parts” of the visit:
– History-taking
– Physical examination
– Counseling

Perform longitudinally
– No need to do it all at once

Agree on performance metrics
with faculty
Summary

Basic clinical skills are important:
so is the need to observe them!
 Observation is a complex skill that
requires training and practice
 Direct observation by educators will
remain a critical component of both
evaluation and feedback