Transcript Selection to the Foundation Programme

Changes in selection to the
Foundation Programme
Professor Paul O’Neill
Chair, ISFP Project Group
Member UKFPO Rules Group
Lead for Research and Evaluation Selection
Plan for Talk
• Background to change – robust & numbers
• Evidence around selection
• SJTs
• Evidence
• Piloting
• EPM
• Algorithm
• Academic FP
Selection (appointment)
Best prediction of the right person to
do the best job
Should be done fairly
PON experience:
• House jobs (Foundation)
• CVs. application forms and letters
• References & ‘putting a word in’
• Interview panels – numbers, questions
• Presentations
• Occupational pyschologists
• Personality testing
Now:
• Mostly national
• Application - anonymous, standardised and
assessed against criteria
• Use of standardised tests (e.g. SJT)
• Selection centres – standardised, multimodal,
some competency testing
• Interview panels more & more standardised
Evolution
2009
DH commissioned a review of selection
methods. The Improving Selection to the
Foundation Programme project was set up
and overseen by the Medical Schools Council
2010/2011 New selection methods were piloted
successfully
FP 2012
Full-scale Parallel Recruitment Exercise (PRE)
FP 2013
New selection methods implemented
A collaborative venture between
Scottish Foundation Board
Foundation School Directors
Educational Supervisors
Selection in Medicine
Undergraduate
• Single School
• UKCAT
Postgraduate
• Foundation
• Postgraduate – Birmingham Review
Substantive – Consultant & GP
Literature Reviews
3 commissioned
 Warwick
 Newcastle
 Durham
Overlapping
Specify methods used
Identify gaps and where judgement will be needed
Durham: McLachlan & Turnbull
• 236 References
• Behaviour predictive of future behaviour
• Conscientiousness significant component of
concerns
Warwick: Thistlethwaite et al
• 197 papers 1990-2009, medical school and
residency.
• Most from USA
• Mostly looked at face validity and reliability NOT
predictive validity
• Lack of consensus
• Non-medical literature – focus on employability
skills (job analysis)
Newcastle: Illing et al
•
202 references
•
Foundation programme – narrow range of applicants,
very few cannot do the job
Selection methods must satisfy their stakeholders
(employers, students etc)
Cognitive tests are moderately predictive of later
cognitive tests
Non-cognitive elements need to be considered to
ensure that a doctor is able to perform well
Assessment centres allow for a range of methods to
be used
•
•
•
•
Selection Methods
Combination of:
• Cognitive ability (academic)
• Non-cognitive
Academic range very narrow
• High ‘justice’
Incorporate measures of big 5 personality applied
to medicine
• Not IQ or general aptitude tests
Selection Methods
Big 5 Model of Personality is predictive of job and
academic performance
•
•
•
•
Extraversion (outgoing, sociable, impulsive)
Emotional stability (calm, relaxed)
Agreeableness (trusting, co-operative, helpful)
Conscientiousness (hardworking, dutiful,
organised)
• Openess to experience (artistic, cultured, creative)
Assessment Criteria
Reliability
Validity
Consistency
Granularity
Longevity
Will the process pick
the ‘best’ applicants
across the UK?
Will the process be
good for many
years?
Assessment Criteria
Educational Impact
Applicant Burden
Clinician Time
Will the process support or
undermine educational
objectives?
How onerous is the
process for
applicants ?
To what extent does the
process distract from
service delivery?
Assessment Criteria
Compliance
Will the process pick
minimise cheating or
malpractice?
Transparency
Can the applicants see
exactly where the
goal posts are?
Fairness (Justice)
Public Opinion
Is there a level playing
field?
Person in the street thinks
that it is fair
Summary of Results
Lower
5-year
cost
Better score
Selection to FP 2013
Situational Judgement Test (SJT)
• SJTs will replace the ‘white space’ application form questions
• This is an invigilated, machine markable test in exam conditions
• The SJT will consist of 70 questions in 2 hours 20 mins
Educational Performance Measure (EPM)
• The EPM will replace the academic quartile scores. The EPM score is
comprised of three elements:
1. medical school performance in deciles
2. additional degrees
3. academic achievements
The EPM and SJT will each be worth 50 points from a 100 point
application score
The case for change
Concern
SJT and EPM
• ‘White space’ questions not sustainable
as a selection tool
• Will become steadily less
discriminatory between eligible
applicants (limited range of new
questions that can be generated)
• SJTs draw upon bank of items to be
available for each application round
•Situations experienced in the Foundation
Programme varied and complex
• New items built incrementally and
continuously against Job Analysis
• ‘White space’ questions non-invigilated
conditions , model answers
concerns about risk of plagiarism and
coaching
• SJTs in invigilated conditions in the UK
(3 national dates)
• Not possible to revise for the SJT
(scenarios complex, answers relate to
judgement rather than knowledge)
The case for change
Concern
• Long-term technical reliability and
validity could be improved
SJT and EPM
• 30 year evidence for reliability of SJTs
• SJT pilots demonstrate the technical
reliability, internal reliability, and validity
for use for FP selection
•Academic quartile system - difficult to
compare fairly between applicants from
different medical schools (not
standardised or subject to quality
assurance across medical schools)
• EPM - standardised framework for
deciles
• Medical schools and students decide
‘basket of assessments’
• Schools will be required to publish their
locally agreed deciles framework, which
will facilitate transparency and quality
assurance from the wider community.
• Deciles fairer to applicants at margins
Situational Judgement Test
What is a Situational Judgement Test?
SJTs are:
• a test of aptitude
• designed to assess the professional attributes expected of a
Foundation doctor
• based on a detailed job analysis of an FY1 doctor
SJT questions assess your judgement by presenting you with
challenging situations you are likely encounter at work during the first
year of an integrated Foundation Programme
Example SJT Questions
There are two question formats:
• Rank the five responses in the most appropriate order
• Choose the three most appropriate responses from eight
You should answer what you ‘should’ do in the scenario
described, not what you ‘would’ do
Example Question 1 – ranking
Mr Reese has end-stage respiratory failure and needs continuous oxygen therapy.
While you are taking an arterial blood gas sample, he confides in you that he knows he
is dying and he really wants to die at home. He has not told anyone else about this as
he thinks it will upset his family, and the nursing staff who are looking after him so well.
Rank in order the following actions in response to this situation (1= most appropriate;
5= least appropriate).
A.
B.
C.
D.
E.
Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital
Reassure Mr Reese that the team will take account of his wishes
Discuss his case with the multi-disciplinary team*
Discuss with Mr Reese's family his wish to die at home
Discuss Mr Reese's home circumstances with his General Practitioner
Answer to Question 1
B. Reassure Mr Reese that the team will take account of his wishes
C. Discuss his case with the multi-disciplinary team*
E. Discuss Mr Reese's home circumstances with his General Practitioner
D. Discuss with Mr Reese's family his wish to die at home
A. Tell Mr Reese that whilst he is on oxygen therapy he will need to stay in hospital
This question is focusing on effective communication with patients. Ensuring that patients’
informed wishes are met in relation to their care is central to your approach to patient care
and this needs to be communicated to the patient in a reassuring manner even in
situations relating to end of life care (B). These wishes should have been sought when
addressing the management plan for Mr Reese and once identified the multidisciplinary
team needs to be made aware of them in order to ensure that as far as possible Mr
Reese’s views in relation to his end of life care are implemented (C). The management of
Mr Reese will require the active involvement of his GP and communication with the GP is
therefore of importance (E). Any decision to discuss Mr Reese’s wishes in relation to his
end of life care with his family can only be made with the full agreement of Mr Reese (D).
It would not be appropriate to give the patient inaccurate information in order to engineer
a different medical pathway (A).
SJTs (Literature Review 77 papers)
• Management, university, police, engineers
• Large scale selection – short-listing
• Construct validity not clearly identified
• Single construct (e.g. ‘practical intelligence’)
• Can be designed to measure differing constructs
• Predictive validity will depend on what criterion is
targeted
• SJT designed to test interpersonal skills will more
likely predict inter-personal orientated performance
SJTs
• Used nationally to select GP registrars and other ‘high
stakes’ occupations
• significant validity in predicting job performance
• incremental validity over methods such as ability tests
and personality questionnaires
• typically relate to general experience and ability, rather
than job-specific knowledge or experience
• tend to show smaller differences between candidate
groups (e.g. based on race) than cognitive ability tests
Job analysis of FY1 doctor
•
•
•
•
•
•
•
•
•
Commitment to professionalism
Coping with pressure
Effective communication
Learning and professional development
Organisation and planning
Patient focus
Problem solving and decision-making
Self-awareness and insight
Working effectively as part of a team
Job analysis of FY1 doctor
Item development
Item writing
workshops
Previously
piloted &
refined
(43 items)
CIT interviews
(78 items)
(89 items)
Review,
concordance,
piloting
Example Question 2 – multiple choice
You have been prescribed codeine for persistent back pain which has become worse in
the last few weeks. You have noticed that during shifts you are becoming increasingly
tired, finding it difficult to concentrate and your performance, as a result, has been less
effective.
Choose the THREE most appropriate actions to take in this situation
A. Ask a colleague to assist with your workload until you finish your codeine
prescription
B. Make an effort to increase the number of breaks during your next shift
C. Stop taking the codeine immediately
D. Make an appointment to see your General Practitioner
E. Seek advice from a specialist consultant about your back pain
F. Arrange to speak with your specialty trainee (registrar)* before your next shift and
make them aware of your situation
G. Seek advice from your clinical supervisor* regarding further support
H. Consider taking some annual leave
Answer to Question 2
D. Make an appointment to see your General Practitioner
F. Arrange to speak with your specialty trainee (registrar) before your next shift
and make them aware of the situation
G. Seek advice from your clinical supervisor regarding further support
This question looks at how you demonstrate commitment to professionalism and selfawareness. The essential problem is that as an FY1 doctor the level of your clinical
performance is dropping. This constitutes a risk to the patients you are caring for and
will impose a greater workload on your colleagues. In this circumstance you should
inform and seek the advice of the senior clinician responsible for your work (G) and
alert your colleagues (F). This matter is most likely to be related to your prescribed
medicine and you should therefore consult with your GP (D) rather than any other
specialist (E). It is not your place to re-allocate workload (A). Increasing the number
of breaks is unlikely to improve a situation that is likely to be due to an adverse effect
of a drug (B). You should not make any unilateral decisions about your medical
treatment (C) and should seek the advice of others (D). You should not be seeking to
use your annual leave (H) to compensate for a medical problem.
Scoring of the SJT
Scoring key, determined through:
• Consensus at item review stage (item writers, SMEs)
• Expert judgement in concordance panel review
• Review and analysis of the pilot data
Scoring not “all or nothing”, but based on how close to scoring key
Scoring of the SJT – ranking
• Up to 20 marks available
• Up to 4 marks available for each response (points for “near misses”)
• No negative marking
Scoring of the SJT – multiple choice
A
B
C
D
E
F
G
H
0
4
0
0
0
0
4
4
• Points for each correct answer
• No negative marking
• 4 points for each correct answer
• Up to 12 points per item
Parallel Recruitment Exercise (PRE)
• New selection methods trialled alongside the normal selection
methods during 2012 FP application round
• Aims: logistics, awareness, pilot new SJT content
• All 31 medical schools involved
• SJT – 1 hr, 30 questions
• EPM – each medical school consulted study body on ‘basket of
assessments’ to be used
Parallel Recruitment Exercise (PRE)
• 90+% of FP applicants participated in the SJT
• Valuable learning experience ahead of live implementation
• Feedback to inform live implementation:
• Applicants
• Medical schools
• Was the final step in ensuring the selection methods can be
consistently and robustly applied for implementation for FP2013
PRE - SJT
30 item, one hour SJT
6,842 medical students took part in the PRE
Participants included:
Sheffield SJT pilot
• Final year medical students
• Students who had been pre-allocated to the Defence Deanery
• Students who had chosen to take a year out post-graduate
• International students returning overseas after graduation
30 medical schools (plus two centres for Eligibility Office applicants)
delivered the SJT in 72 venues
Psychometric analysis shows that a 60 item SJT is a reliable
measurement
Descriptive Statistics
Internal Reliability:
• Adjusted for a 60 - item test that included only robust
items (such as would be used in an operational paper),
all papers had an estimated reliability of α = 0.80 or
above (α = 0.80 to α = 0.87)
• Demonstrates that the SJT is a reliable test in this
context with items testing different things
Histograms showing the distribution of scores for
Paper One (left) and Paper Two (right)
Descriptive Statistics
Mean:
• Overall Mean = 79.5%; Range from 78.0% to 80.6%.
So not too easy – differentiating appropriately
Standard deviations:
• Mean SD = 18.6; Range from 17.3 to 20.0.
Distributions:
• 305 to 468 (out of a maximum of 512) – as expected
given length of paper
• Appears to be slightly negatively skewed, (more people
towards top end) although results do show a close to
normal distribution
Item Facility (difficulty)
Ranked Items
Multiple Choice Items
Maximum score 20
Maximum score 12
Score 18 = ‘very easy’;
Score 10.8 = ‘very easy’;
Score 11.6 = ‘very hard’
Score 3.6 = ‘very hard’
Mean facility similar across
all papers (approx 16)
Mean facility across papers
ranged from
Range of facility values
differed across papers
7.9-9.0
SD range similar for all papers,
except for Paper 1 where one item
had a very high SD
Range of facility values
differed across papers
SD range was similar for all
papers
Student views
The content of the assessment seemed relevant to the Foundation
Programme
The scenario content seemed appropriate for my training level
Educational Performance Measure
Why change to EPM?
• A clear framework with agreed principles used to calculate the EPM,
ensuring that it is fair, transparent and consistent across the schools
of the UK
• Splitting cohorts into deciles rather than quartiles provides a wider
spread of scores, which makes it easier to differentiate between
applicants, and will be more fair for applicants at the margins
• It makes more sense for all the academic components of the
application to be one part of the application
How is the EPM calculated?
Score produced by applicant’s medical school to reflect achievement
and performance compared to rest of cohort
EPM = 3 parts (maximum 50 points):
1. Medical school performance in deciles (34 – 43 points)
E.g. Top 10% = 43; Top 20% = 42; etc
2. Additional degrees (max 5 points)
3. Educational achievements (presentations, prizes and
publications (max 2 points)
Schools have consulted with students about how the decile points for
the EPM will be calculated.
PRE - EPM
• 27 of 30 medical school initiated a new consultation and review of
framework
• (3 schools consult annually and framework aligns)
• Other schools do consult annually – but undertook new consultation with
students for the PRE
• Majority of schools pleased with student engagement, especially amongst
later years
• Benefit of raising awareness with students & staff
• Students felt sense of ownership
Consultation on frameworks
20
18
16
14
12
10
8
6
4
2
0
Meeting
student reps
Committee
Email to all
students
Online survey Open meeting Student reps Convened
consulted working group
PRE EPM – Decile Points
Decile/
Quartile
1st
2nd
3rd
4th
5th
6th
7th
8th
9th
10th
4th
1
1
2
10
13
37
337
680
671
3rd
2nd
1st
3
5
13
52
625
629
348
30
7
2
18
341
649
606
50
19
10
8
2
604
646
334
27
8
2
3
PRE EPM – successes
• All medical schools have agreed a ‘basket of assessments’ in
consultation with students
• All medical schools aligned with EPM framework
• All medical schools calculated EPM deciles, with around 10% in
each decile (some ties)
• All medical schools confident they can calculate EPM deciles in line
with common principles
Academic Foundation Programme
2012 Same timetable as for standard application
Apply to 2 UoA
Have to sit SJT
Appointed – application, EPM, interview (+/-)
If not appointed, then revert to standard process
Algorithms
• Was initially triggered by student preferences
• Unstable, unfairness
• Worse if increasing number applicants/school
• Changed for FP2012
• Now triggered by application score
• Has changed patterns of applications
Selection to the Foundation Programme
– improving and evolving
FP 2005
Foundation Programme introduced
FP 2006
National timetable and application process
FP 2007
Online application – white space & quartiles
2009-2011 Improving Selection to Foundation Programme
FP 2012
Full-scale Parallel Recruitment Exercise (PRE)
FP 2013
New selection methods implemented
The greatest forward step in the baking
industry since bread was wrapped –
Missouri, 1928
‘
What’s Wrong with SJTs
Lawrence Clinical Pharmacology
Wonderful
Useful in
some
situations
Not fit for my dog
More information
UKFPO - www.foundationprogramme.nhs.uk
•
•
•
•
•
FP 2013 Applicant Handbook
Introductory videos
SJT monograph
SJT practice paper
FAQs (FP 2013, SJT, EPM)
Archived ISFP website – www.isfp.org.uk
Any questions?