Transcript Slide 1

PMETB/COPMeD National
Survey of Trainees
PMETB National Survey
of Trainers, 2007.
PMETB and COPMED
National Trainee Survey 2007
Responses from F2 Doctors
Heather Payne¹, Daniel Smith²,
Elisabeth Paice3
¹Wales School of Postgraduate Medical
and Dental Education, Cardiff University
²Postgraduate Medical Education and
Training Board
3London
Postgraduate Deanery, Chair
of COPMED
National Trainee Survey 2007Aims of workshop
•To share collated feedback from Foundation
Trainees in post in 2007
•To identify notable practice, and potential barriers
to achieving improvement
•To explore expectations and implications for local
practice
•To explore use of the PMETB Trainee Survey web
based reporting tool to run local queries
Introduction
•Second annual National Survey of Trainees
•First ever National Survey of Trainers
Methodology
– Item development
– Consultation
Survey launch
Response rates
Analysis
– Launch of the Reporting Tool
– National report
PMETB Training standards
Background
Aim: To assess compliance against PMETB’s Generic Standards
for Training, including the new Standards for Trainers
The National Surveys are one part of PMETB’s quality
assurance framework designed to enable the board to meet its
statutory requirement to maintain and develop the standard of
postgraduate medical education and therefore it must lead to
quality improvement activity.
Background
1. STANDARDS
Includes approval
and action
planning
5. RESPONSES TO
CONCERNS
Range to ensure
patient and/or
trainee safety
4. VISITS
to deaneries
PMETB
QF
2. SHARED
EVIDENCE BASE
includes minimum
data set
3. SURVEYS
Trainees and
trainers
Methodology: National Survey of Trainees
Item development
Last year’s generic items, plus:
– Foundation items
– Psychometric items
– Specialty specific items
Specialty specific items aimed to measure adequate
experience.
Approximately 5 items approved by the SAC chair or
equivalent person. Items consulted on.
Methodology: National Survey of Trainers
Item development
•Literature search to obtain relevant items already in use,
presented to Trainer Survey Working Group for selection
•Consultation conducted of chosen items (deaneries and
colleges/faculties)
•Face validity of items conducted by researchers at NES and
the Northern Deanery
•Final amendments and consultation with working group
members and deanery contacts.
ROCR-lite approval – England only
ROCR-lite (sic) approval was awarded for both surveys. The surveys
have been developed in consultation with The Information Centre for
Health and Social Care who considers the data collection to be useful
and reasonable.
What is ROCR-lite?
ROCR-Lite is a service for Concordat signatories and associated
bodies. The process is designed to:
•allow flexibility in meeting the needs of the various organisations it
seeks to support
•to ensure compliance will differ between organisations.
Survey launch
Trainee Survey - Pilot of administration process in Wales
deanery 15 Nov start; all other deaneries started 4 Dec. The
survey closed on 28 Feb to allow time to chase nonrespondents and missing deanery data.
This year the survey was mandatory for all run-through
trainees, as stated in the Gold Guide.
Trainer Survey - Pilot of the administration in London 14
Dec, all other deaneries 3 Jan. Survey closed 28 Feb.
Both surveys received considerable support from deanery staff
and MEM, who kindly provide the necessary data to administer
them and promoted them to the respondents.
Survey launch process
Trainee Survey – Emails collected from deaneries together
with post data. Emails sent via the database to trainees and
reminders sent to non-respondents.
Trainer Survey – Deaneries provided a nominated contact
such as the DME or equivalent within the LEP. Nominated
contacts were then asked to cascade the survey invitation to
all consultants at their location.
Active GP trainers and Foundation tutors received their
invitation directly from PMETB.
PMETB Trainee survey
Why?
• To assess satisfaction with training
• Quality assurance using PMETB standards
How?
•Survey run Dec 2007- Feb 2008
•All Doctors in UK training posts on 30 Nov
2007
Survey response rates 2007
Overall response rate (all trainees)
66% (n=33,329)
Response rate by Deanery
45-82%
Foundation response rate
56% (n=6,897)
F2 responses n=3,776
Demographic of F2 Trainees 2007
• 58% female
• 91% age 30 or under
• 82% UK graduates
• 5% intended to leave UK once
training complete
At start of post, F2 Trainees…..
Told about roles and
responsibilities
%
Given all the information they
need
%
Identified Educational
Superviser
%
Discussed educational
objectives with Superviser
%
At start of post, F2 Trainees…..
Told about roles and
responsibilities
85%
Given all the information they
need
71%
Identified Educational
Superviser
99%
Discussed educational
objectives with Superviser
82%
F2 Trainees - Induction
•Notable practice
•Barriers to good enough practice
•What needs to change?
F2 Trainees
- Education and Assessment
Using an Educational Portfolio
%
Confident that post would deliver %
required competences
Experience good or excellent
%
F2 Trainees
- Education and Assessment
Using an Educational Portfolio
90%
Confident that post would deliver 75%
required competences
Experience good or excellent
66%
F2 Trainees
- Education and Assessment
•Notable practice
•Barriers to good enough practice
•What needs to change?
Quality of Supervision items
•How often have you felt forced to cope with problems
beyond your competence or experience?
•How often, if ever, have you been supervised by
someone who you feel isn’t competent to do so?
•How often have you been expected to obtain consent
for procedures which you do not carry out yourself?
•Do you always know who is providing your clinical
supervision when you are working?
•Please indicate your perception of the way in which
critical incidents and near misses are reported in your
department
F2 Trainees
- Supervision and Support
Formal meeting with
superviser on performance in
post
%
Receiving informal feedback
from senior at least weekly
%
Formal assessment of
performance in workplace
%
F2 Trainees
- Supervision and Support
Formal meeting with
superviser on performance in
post
50%
Receiving informal feedback
from senior at least weekly
34%
Formal assessment of
performance in workplace
48%
F2 Trainees
- Supervision and Support
•Notable practice
•Barriers to good enough practice
•What needs to change?
F2 Trainees - Career influences
Discussed career plans with
senior
%
Decided on career intention
%
Would recommend post to a
friend
%
Feel post not useful in career
%
F2 Trainees - Career influences
Discussed career plans with
senior
71%
Decided on career intention
91%
Would recommend post to a
friend
67%
Feel post not useful in career
10%
F2 Trainees - Career influences
•Notable practice
•Barriers to good enough practice
•What needs to change?
F2 Trainees - Safe Doctors?
Critical event/ near miss
reporting encouraged and
followed up
Coping beyond competence
weekly or daily
%
%
F2 Trainees - Safe Doctors?
Critical event/ near miss
reporting encouraged and
followed up
73%
Coping beyond competence
weekly or daily
32%
Medical Errors: survey questions
7. In the last month, have you made a serious medical
error?
No / Yes, once / Yes, more than once /Do not wish to answer
8. In the last month, have you made a potentially
serious medical error?
No / Yes, once / Yes, more than once /Do not wish to answer
UK trainees reporting errors
‘in the last month’
Foundation compared to all other grades
No
Yes,
once
Yes,
more
than
once
N
In the last month, have
you made a serious
medical error?
All other
training grades
98.3%
1.6%
0.1%
25,986
Foundation
96.3%
3.3%
0.4%
6,549
In the last month, have
you made a potentially
serious medical error?
All other
training grades
91.1%
8.3%
0.5%
25,876
Foundation
81.5%
16.7%
1.7%
6,485
F2 Trainees - Safe Doctors?
•Notable practice
•Barriers to good enough practice
•What needs to change?
F2 Trainees - satisfaction
Felt subject to persistent
behaviour undermining
confidence and self esteem
%
Considered daily leaving
medicine
%
F2 Trainees satisfaction
Felt subject to persistent
behaviour undermining
confidence and self esteem
10%
Considered daily leaving
medicine
9%
Source of bullying
– trainees who reported bullying only
Othr
Cons T’ee
Nrs
Pt/ Wk
Mgr rel cult
50%
11%
12%
4%
1% 12% 7%
3%
2,247
26%
26%
25%
2%
2% 12% 5%
2%
734
Othr MW N=
All
othr
gde
Fou
ndat
ion
F2 Trainees - Satisfaction
•Notable practice
•Barriers to good enough practice
•What needs to change?
F2 Trainees
Take Home Messages
•Notable Practice
•Potential barriers
F2 Trainees
Notable Practice Action Plan
•Action Points for my workplace
•Useful contacts and resources
Analysis
Response rates
Trainee
•All usable responses = 33,329,
•Response rate = 66% where denominator is all valid trainee
emails obtained via deanery returns and trainee opt-in form.
Trainer
•GP = 2,211 (44%)
•Consultant = 7,810 (~20%)
•Total = 10,021
Analysis
The Reporting Tool
•Launched to deanery staff involved in the work 2 May for
testing and comment prior to public launch.
•The reporting tool for the survey results is located here:
http://reports.pmetb.org.uk
•Each type of report covers a different type of
respondent group and is displayed via three levels of
data, starting with the top level and going down to the
lowest detail of individual items.
•Each level can be accessed by clicking through or
selecting an option from the drop down list.
How To Use The Reporting Tool
1) After logging in, select the Group type you wish to view.
Select the reporting
group you are
interested in. This
includes selecting the
survey type and year.
How To Use The Reporting Tool
2) Next refine your data selection, for example select the
deanery and specialty of interest.
To view the data for
your deanery and
specialty select the
appropriate options in
the drop-down lists.
How To Use The Reporting Tool
3) View each indicator score by scrolling down the page and note
the national range in the green box, values will be shown when
you roll your mouse over them.
To view the next level
of data please click
here.
How To Use The Reporting Tool
4) The next screen
displays mean
indicator values
within the selected
group of providers.
To view the next level
of data click on the
yellow dot (the selected
deanery) or select the
deanery from the dropdown below.
To view the deanery
labels roll over each
point
How To Use The Reporting Tool
To view the data labels
for the bar charts roll
you mouse over each
bar.
5) The last screen
displays individual
items and also
displays the N
number. There is a
select box on this
page at the bottom
to view other groups
of items at this
level.
Trainee Survey example
All Trauma and orthopaedic surgery UK-wide 2007
This example shows a outlier below the national mean and in
the lower quartile. In this case trauma and orthopaedic
surgery F1, F2, FTSTA and Lower (ST1-3) trainees have
outlier status for the overall supervision indicator score when
compared to all trainees UK-wide.
Trainee Survey example
All Trauma and orthopaedic surgery UK-wide F2 2007
By clicking through
to the next level
you can see that
trauma and
orthopaedic F2
score the lowest
when compared to
other surgical
specialties for
overall supervision.
Trainee Survey example
All Trauma and orthopaedic surgery UK-wide F2 2007
Overall Supervision
The bottom level
graphs show the items
used to calculate the
indicator. In this case
the poor indicator
score is partly
attributed to 40% of
respondents selecting
‘weekly’ to the
statement shown on
the right.
N = 240
Trainer Survey example
South Yorkshire & South Humber 2007
This indicator is the percentage of those trainers who perform
workplace based assessments who have been trained to perform
them. A high score indicates more trainers having received training
in workplace based assessments out of those respondents who say
they undertake this work.
This example shows that the SYSH deanery have an above outlier,
indicating that the mean score for these trainers is above the
national mean.
Trainer Survey example
Comparison across selected group of providers. South Yorkshire &
South Humber
All deaneries. Workplace based assessments (%)
The next level
shows that the
SYSH deanery has
the highest % of
workplace based
assessment trained
trainers in the UK.
Trainer Survey example
South Yorkshire & South Humber 2007
Feedback To Trainees (%)
The bottom level
graphs show the items
used to calculate the
indicator. In this case
the high indicator
score is attributed to
75% of respondents
stating that they have
been trained in
workplace based
assessments.
N = 210
National report
•The National reports for the Trainee and Trainer
Surveys, 2007 will be launched week commencing
21 July 2008
•Deaneries will respond to survey findings via the
action plans contained within the deanery annual
reports
•Launch event will be held in Cardiff 25 July 2008.
Further information
For additional information please
see the PMETB website:
www.pmetb.org.uk/pmetbsurveys
If you have any comments or
queries please contact:
[email protected]
Or
[email protected]