Evidence Informing Practice

Robert Coe ASCL Annual Conference, 21 March 2014

 

Outline

What can research tell us about the likely impacts and costs of different strategies?

How do we implement these strategies to … – Focus on what matters – Target areas of need – Produce demonstrable benefits Improving Education: A triumph of hope over experience http://www.cem.org/attachments/publications/ImprovingEducation2013.pdf

2

Evidence about the effectiveness of different strategies

3

Toolkit of Strategies to Improve Learning ∂ The Sutton Trust-EEF Teaching and Learning Toolkit http://www.educationendowmentfoundation.org.uk/toolkit/

www.educationendowmentfoundation.org.uk/toolkit

Impact vs cost

Most promising for raising attainment

8 Feedback Meta-cognitive

May be worth it

Peer tutoring 0 Homework (Secondary) ∂ Collaborative Behaviour Phonics Small gp tuition Social Homework ICT Individualised learning Parental involvement Summer schools Mentoring 1-1 tuition £0 (Primary) Ability grouping Performance pay Aspirations Cost per pupil Teaching assistants After school £1000 Early Years Smaller classes

Small effects / high cost

Key messages

 Some things that are popular or widely thought to be effective are probably not worth doing  Performance pay; Raising aspirations Some things look ‘promising’ – Effective feedback; Meta-cognitive and self regulation strategies; Peer tutoring/peer ‐ assisted learning strategies; Homework

Clear, simple advice:

  Choose from the top left Go back to school and do it ∂ For every complex problem there is an answer that is clear, simple, and wrong H.L. Mencken 7

 

Why not?

We have been doing some of these things for a long time, but have generally not seen improvement Research evidence is problematic – Sometimes the existing evidence is thin  – Context and ‘support factors’ may matter Implementation is problematic – We may think we are doing it, but are we doing it right?

– We do not know how to get large groups of teachers and schools to implement these interventions in ways that are faithful, effective and sustainable 8

So what should we do?

9

Four steps to improvement

    Think hard about learning Invest in good professional development Evaluate impact of changes

1. Think hard about learning

www.educationendowmentfoundation.org.uk/toolkit

Impact vs cost

Most promising for raising attainment

8 Feedback Meta-cognitive

May be worth it

Peer tutoring 0 Homework (Secondary) ∂ Collaborative Behaviour Phonics Small gp tuition Social Homework ICT Individualised learning Parental involvement Summer schools Mentoring 1-1 tuition £0 (Primary) Ability grouping Performance pay Aspirations Cost per pupil Teaching assistants After school £1000 Early Years Smaller classes

Small effects / high cost

1. Which strategies/interventions are very

surprising

(you really don’t believe it)?

2. Which strategies/interventions can you

explain why

they do (or don’t) improve attainment?

3. Which strategies/interventions o you

want to know more

about?

13

     

Poor Proxies for Learning

Students are busy: lots of work is done (especially written work) Students are engaged, interested, motivated Students are getting attention: feedback, explanations Classroom is ordered, calm, under control Curriculum has been ‘covered’ (ie presented to students in some form) (At least some) students have supplied correct answers, even if they – Have not really understood them – Could not reproduce them independently – Will have forgotten it by next week (tomorrow?) – Already knew how to do this anyway 14

Do children learn better in

∂

the morning or afternoon?

A better proxy for learning?

Learning happens

∂

when people have to think hard

Hard questions about your school

 How many minutes does an average pupil on an average day spend really thinking hard?

  in your lessons?

If they knew the right answer but didn’t know why, how many pupils would care?

17

2. Invest in effective CPD

How do we get students to learn hard things?

Eg  Place value  Persuasive writing  Music composition  Balancing chemical equations • Explain what they should do • Demonstrate it • Get them to do it (with gradually reducing support) • Provide feedback • Get them to practise until it is secure • Assess their skill/ understanding

How do we get teachers to learn hard things?

Eg  Using formative assessment  Assertive discipline  How to teach algebra • Explain what they should do ∂

What CPD helps learners?

     

Intense

: at least 15 contact hours, preferably 50

Sustained

: over at least two terms

Content focused

: on teachers’ knowledge of subject content & how students learn it ∂

Active

: opportunities to try it out & discuss

Supported

: external feedback and networks to improve and sustain

Evidence based

: promotes strategies supported by robust evaluation evidence

3. Evaluate teaching quality

    

Why monitor?

Strong evidence of (potential) benefit from – Performance feedback (Coe, 2002) – Target setting (Locke & Latham, 2006) – Intelligent accountability (Wiliam 2010) Individual teachers matter most Everyone can improve ∂ Teachers stop improving after 3-5 years Judging real quality/effectiveness is very hard – Multidimensional – Not easily visible – Confounded 23

   

Monitoring the quality of teaching

Progress in assessments – Quality of assessment matters ( cem.org/blog ) – Regular, high quality assessment across curriculum ( InCAS , INSIGHT ) Classroom observation – Much harder than you think! ( cem.org/blog ) – Multiple observations/ers, trained and QA’d Student ratings ( ∂ – Extremely valuable, if done properly http://www.cem.org/latest/student-evaluation-of-teaching-can it-raise-attainment-in-secondary-schools ) Other – Parent ratings feedback – Student work scrutiny – Colleague perceptions (360) – Self assessment – Pedagogical content knowledge 24

  

Teacher Assessment

How do you know that it has captured understanding of key concepts?

– vs ‘check-list’ (eg ‘;’=L5, 3 tenses=L7) How do you know standards are comparable?

∂ – Across teachers, schools, subjects – Is progress good?

How have you resolved tensions from teacher judgments being used to judge teachers?

25

Evidence-Based Lesson Observation

   Behaviour and organisation – Maximise time on task, engagement, rules & consequences Classroom climate – Respect, quality of interactions, failure OK, high expectations, growth mindset Learning – What made students think hard?

– Quality of: exposition, demonstration, scaffolding, feedback, practice, assessment – What provided evidence of students’ understanding?

– How was this responded to? (Feedback) 26

Next generation of CEM systems …

   Assessments that are – Comprehensive, across the full range of curriculum areas, levels, ages, topics and educationally relevant abilities – Diagnostic, with evidence-based follow-up – Interpretable, calibrated against norms and criteria – High psychometric quality ∂ Feedback that is – Bespoke to individual teacher, for their students and classes – Multi-component, incorporating learning gains, pupil ratings, peer feedback, self evaluation, … – Diagnostic, with evidence-based follow-up Constant experimenting 27

4. Evaluate impact of changes

School ‘improvement’ often isn’t

  School would have improved anyway – Volunteers/enthusiasts improve: misattributed to intervention – Chance variation (esp. if start low) Poor outcome measures – Perceptions of those who worked hard at it   Poor evaluation designs – Weak evaluations more likely to show positive results – Improved intake mistaken for impact of intervention Selective reporting – Dredging for anything positive (within a study) – Only success is publicised

(Coe, 2009, 2013)

Key elements of good evaluation

∂  Clear, well defined, replicable intervention  Good assessment of appropriate outcomes  Well-matched comparison group

Summary …

1. Think hard about learning 2. Invest in good CPD 4. Evaluate impact of changes [email protected]

www.cem.org @ProfCoe