Transcript Document

How do you choose a washing
machine? Using evidence to inform
practice
Dr Jonathan Sharples
www.educationendowmentfoundation.org.uk
The EEF approach
Summarise
the existing
evidence
Share and
promote
the use of
evidence
Make
grants
Evaluate
projects
“We must give educators and
politicians the information they
need to make wise decisions for
children”
Estelle Morris
• Stand on the shoulders of previous progress i.e. healthcare, engineering
• Research is seen as something done to, not with, or for, or by the
profession – culture change
What is evidence-informed practice?
How do we generate useful evidence for
practice?
What can be done to make research more
accessible to busy professionals?
How might we build capacity in professional
settings to use research evidence at scale?
Make abstract concepts a bit more
tangible...
www.alliance4usefulevidence.org
Current moves towards more evidence-informed policy/practice…
Royal College of Teachers
ResearchED
Cabinet Office
‘What works’
centres
Need for good
evidence is
increasingly agreed
Weak link is the coupling
of evidence to practice, at
scale –
‘Knowledge
mobilisation’
Challenges – Fragmentation in the system, weak links to training, OfSTED?
What are the challenges in accessing and using
reliable research evidence?
Knowing where to look for useful
information
Sifting reliable research conclusions
from the rest
Time, time, time!
Senior Leadership Teams
Too much information is
available
Can be a conflict with
existing school
practices
Not skilled to judge the validity
of claims
www.cebenetwork.org
Practitioners
Researchers
Policy makers
What would an evidence-using and creating culture
look like?
What do we need to get there?
2013/14 - Events series on knowledge
mobilisation in practice
The EEF by numbers
3,000
schools
participating
in projects
34
topics in
the Toolkit
14
6,000
members
of EEF
team
600,000
pupils involved
in EEF projects
heads
presented to
since launch
£220
16
m
estimated spend
over lifetime of
the EEF
10
reports
published
independent
evaluation
teams
83
evaluations
funded to
date
Applying evidence in practice
Step 1: Decide what you want to achieve
Identify school priorities using internal data
and professional judgement.
Step 5: Securing and spreading change
Mobilise the knowledge and use the findings
to inform the work of the school to grow or stop
the intervention.
Step 4: Did it work?
Evaluate the impact of your decisions and
identify potential improvements for the future.
Step 2: Identifying possible solutions
External evidence summarised in the Toolkit
can be used to inform choices.
Step 3: Giving the idea the best chance of success
Applying the ingredients of effective
implementation.
10
Applying evidence in practice
Step 1: Decide what you want to achieve
Generate a question using data,
professional judgement and values.
How can we engage and support
struggling readers in Year 7?
Applying evidence in practice
Step 2: Identify possible solutions
Ensure that you start from the best position
by seeking internal and external
knowledge.
What has been shown to be effective in
raising outcomes for struggling readers?
A Pupil Premium Scenario –
Struggling Readers
Number on Roll
%FSM
Pupil Premium
Allocation
750
10%
£67,500
Parents
Class size
reduction
Head of English
One to one tuition
SENCO
Employ more
Teaching Assistants
What do you decide to spend the money on?
How do you make the most of that investment?
Teaching and Learning Toolkit
• The Toolkit is an accessible, teacher-friendly summary of
educational research. ‘Which?’ for education
• Practice focused: tries to
give schools the
information they need to
make informed decisions
and narrow the gap.
• Based on meta-analyses
conducted by Durham
University.
Effective classroom
strategies for closing the
gap in educational
achievement for
children and young
people from poor
backgrounds, including
white working class
boys
Teaching and Learning Toolkit
• The Toolkit is an accessible, teacher-friendly summary of
educational research. ‘Which?’ for education
• Practice focused: tries to
give schools the
information they need to
make informed decisions
and narrow the gap.
• Based on meta-analyses
conducted by Durham
University.
The Toolkit is a starting point for
making decisions
Overview of value for money
10
Effect Size (months gain)
Feedback
Meta-cognitive
Peer tutoring
Homework
Phonics
Learning Individualised
learning
styles
Arts
Ability grouping
0
£0
Independent learning
Pre-school
1-1 tutoring
Outdoor
ICT
learning
After
Parental
school
involvement
Summer
schools
Sports
Performance
pay
Cost per pupil
Smaller
classes
Teaching
assistants
£1000
Using the Toolkit
Use the evidence as a starting
point for discussion.
Dig deeper into what the
evidence actually says
Understand the ‘active
ingredients’ of implementation
Example: Teaching Assistants
Better magazine and Best Evidence in Brief
Three/year. Free for first year –
www.betterevidence.org
Free fortnightly research digest –
[email protected]
Applying evidence in practice
Step 3: Give the idea the best chance of success
Implementation matters: have you thought
about what the approach means for teaching and
learning?
What are the ‘active ingredients’ for making best
use of teaching assistants for struggling readers?
How much training do
TA’s need?
?
How will you organise the
tuition during classtime?
Is there disruption to other learning?
Implementation matters: how is as
important as what the evidence says
In the US, healthcare workers failure to wash hands effectively is major
cause of death – $billions
Researcher created a checklist for surgical teams. Trial showed 66%
reduction in infection rates, ~1500 lives in 18 months. Packaged the
principles of handwashing into a practical intervention.
Shortage of practical vehicles (interventions, CPD training) to help get
evidence working in practice, at scale and with rigour (eg AfL)
EEF Projects
We are working to fund, develop and
evaluate projects that:
• Build on existing evidence.
• Will generate significant new
understanding of ‘what works’.
• Can be replicated cost effectively if
proven to work.
Examples: One-to-one support with
teaching assistants, lesson observation,
using mobile devices for feedback,
Switch on Reading
• One to one literacy intervention with children in Year 7 who are struggling
with literacy (not achieving level 4 at KS2)
• Based on Reading Recovery. Delivered by teaching assistants,
20mins/day over 10 weeks.
• Previous research shows a positive effect (inc. Reading Recovery).
• RCT in 19 schools with 300 pupils
• Attainment measured using
standardised literacy measures
• Independent evaluation by Durham
University
• Observations and interviews to inform
how and why the approach might be
working
http://educationendowmentfoundation.org.uk/projects/category/primary
Switch On Reading
Number of
pupils
Effect size*
Estimated
months’
progress
All pupils
308
+0.24
+3
Lower
attainers
156
+0.39
+5
FSM-eligible
98
+0.36
+4
SEN reported
225
+0.31
+4
Group
Switch On Reading - Conclusions
Key Conclusions
1. Switch-on Reading appears to be effective for weak and
disadvantaged readers at the stage of transition to secondary school.
2. It can be delivered by teaching assistants after two-days of training,
and full training and support is required for all relevant staff.
3. Challenges to successful implementation may include timetabling
and the availability of age-appropriate texts.
4. There is a tendency for some staff to stray away from the explicit
schedule and this is likely to reduce the programme’s impact. Regular
monitoring will increase fidelity but may also increase cost.
Catch Up Numeracy
• One to one intervention with children in Years 2 to 6 who
are struggling with numeracy
• Two 15 minute sessions with TAs per week for 30 weeks
Number of
pupils
Effect size
Estimated
months’
progress
Catch Up Numeracy
vs. control
108
+0.21
+3
Equivalent time one
to one support vs.
control
102
+0.27
+4
Group
Catch Up Numeracy Conclusions
Key Conclusions
1. Within this trial, one-to-one support by TAs led to a significant gain
in numeracy skills.
2. Catch Up Numeracy makes similar significant gains, but there is
little evidence that it provided any additional gains in numeracy
outcomes over and above those from one to one teaching itself.
3. Schools can find it challenging to run two 15 minutes sessions per
week, due to timetabling and other issues.
4. Structured interventions, such as Catch Up Numeracy, should be
planned into the timetable from the start of the new school year to
ensure they are given priority and status.
What have we learned?
• Teaching assistants, given the right support and training,
can make a significant contribution to pupil attainment
• Schools should take care to understand how specific
programmes are having an impact in their school.
• Implementation matters:
brief, well-structured 1-2-1
sessions over a sustained
period, with appropriate
support and training
Not all positive findings though….
IEE Evidence for Impact database…
coming soon
Applying evidence in practice
Step 4: Put energy into evaluation
Did the approach work, what made it work,
and how can it be improved next time?
Can we demonstrate that our
readers are making progress?
Is it worth the effort?
The Toolkit is a starting point for
making decisions
Overview of value for money
Promising
10
May be
worth it
Effect Size (months gain)
Feedback
Meta-cognitive
Peer tutoring
Homework
Phonics
Learning Individualised
learning
styles
Arts
Ability grouping
0
£0
Independent learning
Outdoor
ICT
learning
After
Parental
school
involvement
Pre-school
1-1 tutoring
Summer
schools
Sports
Performance
pay
Cost per pupil
Teaching
assistants
Smaller
classes
Requires careful
consideration
£1000
Using the Toolkit
Use the evidence as a starting
point for discussion.
Dig deeper into what the
evidence actually says
Understand the ‘active
ingredients’ of implementation
Example: Teaching Assistants
Better magazine and Best Evidence in Brief
Three/year. Free for first year –
www.betterevidence.org
Free fortnightly research digest –
[email protected]
Applying evidence in practice
Step 3: Give the idea the best chance of success
Implementation matters: have you thought
about what the approach means for teaching and
learning?
What are the ‘active ingredients’ for making best
use of teaching assistants for struggling readers?
How much training do
TA’s need?
?
How will you organise the
tuition during classtime?
Is there disruption to other learning?
Implementation matters: how is as
important as what the evidence says
In the US, healthcare workers failure to wash hands effectively is major
cause of death – $billions
Researcher created a checklist for surgical teams. Trial showed 66%
reduction in infection rates, ~1500 lives in 18 months. Packaged the
principles of handwashing into a practical intervention.
Shortage of practical vehicles (interventions, CPD training) to help get
evidence working in practice, at scale and with rigour (eg AfL)
Switch on Reading
• One to one literacy intervention with children in Year 7 who are struggling
with literacy (not achieving level 4 at KS2)
• Based on Reading Recovery. Delivered by teaching assistants,
20mins/day over 10 weeks.
• Previous research shows a positive effect (inc. Reading Recovery).
• RCT in 19 schools with 300 pupils
• Attainment measured using
standardised literacy measures
• Independent evaluation by Durham
University
• Observations and interviews to inform
how and why the approach might be
working
http://educationendowmentfoundation.org.uk/projects/category/primary
Switch On Reading
Number of
pupils
Effect size*
Estimated
months’
progress
All pupils
308
+0.24
+3
Lower
attainers
156
+0.39
+5
FSM-eligible
98
+0.36
+4
SEN reported
225
+0.31
+4
Group
Switch On Reading - Conclusions
Key Conclusions
1. Switch-on Reading appears to be effective for weak and
disadvantaged readers at the stage of transition to secondary school.
2. It can be delivered by teaching assistants after two-days of training,
and full training and support is required for all relevant staff.
3. Challenges to successful implementation may include timetabling
and the availability of age-appropriate texts.
4. There is a tendency for some staff to stray away from the explicit
schedule and this is likely to reduce the programme’s impact. Regular
monitoring will increase fidelity but may also increase cost.
Catch Up Numeracy
• One to one intervention with children in Years 2 to 6 who
are struggling with numeracy
• Two 15 minute sessions with TAs per week for 30 weeks
Number of
pupils
Effect size
Estimated
months’
progress
Catch Up Numeracy
vs. control
108
+0.21
+3
Equivalent time one
to one support vs.
control
102
+0.27
+4
Group
Catch Up Numeracy Conclusions
Key Conclusions
1. Within this trial, one-to-one support by TAs led to a significant gain
in numeracy skills.
2. Catch Up Numeracy makes similar significant gains, but there is
little evidence that it provided any additional gains in numeracy
outcomes over and above those from one to one teaching itself.
3. Schools can find it challenging to run two 15 minutes sessions per
week, due to timetabling and other issues.
4. Structured interventions, such as Catch Up Numeracy, should be
planned into the timetable from the start of the new school year to
ensure they are given priority and status.
What have we learned?
• Teaching assistants, given the right support and training,
can make a significant contribution to pupil attainment
• Schools should take care to understand how specific
programmes are having an impact in their school.
• Implementation matters:
brief, well-structured 1-2-1
sessions over a sustained
period, with appropriate
support and training
IEE Evidence for Impact database…
coming soon
Applying evidence in practice
Step 4: Put energy into evaluation
Did the approach work, what made it work,
and how can it be improved next time?
Can we demonstrate that our
readers are making progress?
Is it worth the effort?
Use DIY guide to evaluate innovations
A traditional school approach to
intervention impact
29 students
receive an
intervention to
improve their
attainment
Year 9 Class
Year 9 Class
29 students
29 students
Data indicates low
rates of attainment
Data indicates 32% of
them increased their
levels
Conclusion?
Oral feedback was
responsible for the
rise in attainment
Control vs. Treatment
Stages of the guide
Applying evidence in practice
Step 5: Making innovation stick
Moving from what we know to what we do.
Have we captured and embedded effective
small group tuition in our school?
Could it make an impact in other areas?
Thank you!
Taking part in EEF research:
[email protected]
[email protected]