Transcript SEFCE South-East Scotland Faculty of Clinical Educators

Work-Place Based Assessment
Dr Rob Waller
Dr Simon Edgar
Associate Director of Medical Education
St Johns Hospital
Director of Medical Education
NHS Lothian
Session Content
•
Description of Activity
– This will be an active workshop in
which we will discuss workplacebased assessment.
– We will focus on the types of
workplace based assessment
rolled out in undergraduate and
post graduate medical education
in the UK such as mini-Clinical
Examination (CEX), multi-source
feedback (MSF) and Direct
Observation of Practical Skills
(DOPS).
– Some materials will be presented,
but most of our time will be spent
in active learning. We will also
discuss methods of feedback in
the context of work-place based
assessment.
•
Learning Outcomes
– Be able to discuss the rationale
for workplace-based assessment
– Understand different types of
workplace-based assessment
– Have reviewed and evaluated
several types of workplace based
assessment
– Be able to develop and test
anchor statements
– Have reviewed feedback in the
context of workplace-based
assessment
– Have considered how to
implement workplace-based
clinical skills assessments in their
main clinical area and how to give
focused feedback to learners
Timeline
•
•
•
•
•
•
•
900:
945:
1000:
1030:
1100:
1130:
1200:
Intro and Rationale
Types of WPBA
Anchor statements
Break for refreshments
Evaluating their usefulness
Giving good feedback
Local implementation
Preparation
• Small Groups
– Think about your prior experiences of
workplace based assessment and feedback
to share with the group
• Definition of WPBA [5]
• Good examples and Bad examples [10]
Rationale
• Historically an apprenticeship model
– Unstructured, holistic/subjective judgements
– Risk of halo/hawthorn effect, ‘old boys network’
– But did it have high face validity?
• Now outcome/competency based
– Structured, objective assessment
– Competency ‘in the clinical setting’ = Performance
• Challenges
– Making the feedback good [this is the bit that works]
– Reliability, validity, acceptability [student/teacher]
– Relative paucity of evidence base
• Further Resources
– http://www.faculty.londondeanery.ac.uk/e-learning/workplacebased-assessment
Reflection
• Small groups [10]
– Can you think of some professional competencies
that are difficult to assess within traditional
assessments, e.g. multiple-choice tests, written
papers, OSCEs, etc? GMC Good Medical Practice
– Can you identify some ways in which you might
assess one of these competencies using information
derived directly from the doctor’s working
environment?
– What might be the advantages or disadvantages of
such an assessment?
Literature
• Norcini & Burch, Med Teach 2007; 29(9):855-871 [PDF]
– The advantage of being timely and specific
– Good faculty essential – training, good feedback skills
• Miller & Archer, BMJ 2010; 341, Sys Rev [PDF]
– 16 studies, only one RCT, 8 on MSF
– Kirkpatrick: 2 at level 2, 3 at level 4
– MSF the only type with any real benefit
• Jacques, BMJ Careers, 30 Nov 2011
– RCGP Review of three year programme
– Little or no correlation of WPBA to final outcome
– Submitted [successful] business case for 4 year training
• Burr et al, BMJ Careers, 31 Jul 2012
– Reducing the number of assessments and increasing free text
responses
– The move towards the SLE – supervised learning event
Types of WPBA
• Five areas
–
–
–
–
–
Clinical Encounter [Mini-CEX, DOPS/DONCS]
Discussion of Clinical Skills [CBD]
Multi-source Feedback [TAB, 360] [Patient?]
Performance Data [Logbook] [Clinical Outcomes]
Developing the Clinical Teacher
• Small groups
– Pick an area
– Describe the relevant WPBA
– What would constitute competency?
Anchor Statements
• These guide marking to the appropriate level
– Trainee at current stage vs end of training
– Novice, advanced beginner, competent, proficient,
expert 1
– RCGP Statements, Blooms Taxonomy
• Small Group:
– Write anchor statements for the following:
• Competency: Able to delegate effectively
• Level: Completion of Foundation Programme
• Blank anchor statement
1 Dreyfus, H L and Dreyfus, SE (1986) Mind over
Machine: the power of human intuition and expertise
in the age of the computer, Oxford, Basil Blackwell
RCGP Statements
Example
MRCGP
Anchor
Statemen
ts
Cognitive
Learning
Blooms
Taxonomy
Break
Any questions?
Evaluation of WPBA
• Limited evidence base, but necessary in some
for to give a modern assessment of clinical
‘performance’
– Illustrated by the principles of constructive alignment
and blueprinting
• Importance of triangulation with other data in the
wider portfolio to contribute to end of placement
and programme reviews
– Their role in global assessment [unstructured?]
• Their value in the struggling trainee
– Formative assessment
– Evidence of failure [rarely completed…]
Feedback
• The key is good feedback
• Small groups [15]
– What activities are you currently involved in that require you to
give feedback to trainees or students?
– Have you had any training in giving feedback and, if so, what are
some of the key principles you remember?
– What are your strengths in giving feedback?
– What aspects of giving feedback are you hoping to improve?
• Tension between
– Formative and summative assessment
– Assessment and appraisal
– Who does the assessment? Who gives the feedback?
• Further Resources:
– http://www.faculty.londondeanery.ac.uk/e-learning/feedback/
Good feedback
• Turns experience into
reflection
• Blends with assessment
– A good thing?
• Formal and informal
• Multiple sources
– Issues with this?
• Individual Reflection
– Review exercise
– Review in pairs
Barriers to Feedback
• Small groups [10]
•
Hesketh and Laidlaw 2002
–
– What are the barriers
to effective feedback?
– What can be done to
overcome these?
–
–
–
–
–
–
•
Hesketh EA and Laidlaw JM (2002) Developing
the teaching instinct: feedback. Medical
Teacher. 24: 245–8
Parsloe E (1995) Coaching, Mentoring and
Assessing: A Practical Guide to Developing
Competence. Kogan Page Ltd, London
Parsloe 1995
–
–
•
a fear of upsetting the trainee or
damaging the trainee–doctor
relationship
a fear of doing more harm than good
the trainee being resistant or defensive
when receiving criticism. Poor handling
of a reaction to negative feedback can
result in feedback being disregarded
thereafter
feedback being too generalised and not
related to specific facts or observations
feedback not giving guidance on how
to rectify behaviour
inconsistent feedback from multiple
sources
a lack of respect for the source of
feedback.
Sensitively and appropriately
Acknowledges power imbalances
Personal factors
BMJ: ABC of Learning and
Teaching in Medicine
• Learners value feedback highly, and valid feedback is based on
observation. Deal with observable behaviours and be practical,
timely, and concrete. The one to one relationship enables you to
give feedback with sensitivity and in private. Begin by asking the
learner to tell you what he or she feels confident of having done well
and what he or she would like to improve. Follow up with your own
observations of what was done well (be specific), and then outline
one or two points that could help the student to improve.
• Just as many learning opportunities are wasted if they are not
accompanied by feedback from an observer, so too are they wasted
if the learner cannot reflect honestly on his or her performance. One
to one teaching is ideally suited to encouraging reflective practice,
because you can model the way a reflective practitioner behaves.
Two key skills are (a) ‘unpacking’ your clinical reasoning and
decision making processes and (b) describing and discussing the
ethical values and beliefs that guide you in patient care.
Gordon J (2003) BMJ ABC of Learning and
Teaching in Medicine: one to one teaching and
feedback. British Medical Journal. 326: 543–5
Practice giving feedback
• Bad session
– Vimeo, .m4v
• Small groups [10]
– How would you give
feedback to
• The junior?
• The senior?
• How can you deal with
the Hawthorn effect?
• How can you deal with
Hawks?
• Good session
– Vimeo, .m4v
• Small groups [10]
– How would you give
feedback to
• The junior?
• The senior?
• How can you deal with
the Halo effect?
• How can you deal with
Doves?
The one minute preceptor model
• Anyone can do this!
• Small groups
– Watch this short video – what do you notice? [vimeo]
[.m4v]
• Basic Techniques
–
–
–
–
Open questions
Quick coverage of gaps
Mini-teach
Feedback
• Lick and Scratch – and another Lick…
http://gpst.mvm.ed.ac.uk/resource/769
Implementation
• Barriers
– Lack of belief in value of task
– Busy clinical workload
– Burnt Out clinician / Easier to do it myself
• Small Groups [10]
– For a busy OBGYN Unit with 20 junior doctors, how
many PAs are needed for
• Educational Supervision [assume 3 meetings per 6/12]
• Clinical Supervision [assume 8 WPBA per 6/12]
– How many WTE is this? How is it best delivered?
Summary
Any questions?
Work-Place Based Assessment
Dr Rob Waller
Dr Simon Edgar
Associate Director of Medical Education
St Johns Hospital
Director of Medical Education
NHS Lothian