Transcript Inside the black box: Raising standards through classroom

Why is getting good schools so
hard?
Dylan Wiliam
September 2010
www.dylanwiliam.net
Overview: science and design




We need to improve student achievement
This requires improving teacher quality
Improving the quality of entrants takes too long
So we have to make the teachers we have better
Science
 We can change teachers in a range of ways
 Some will benefit students, and some will not.
 Those that do involve changes in teacher practice
 Changing practice requires new kinds of teacher
learning
 And new models of professional development.
Design
Raising achievement matters…
For individuals
• Increased lifetime salary
• Improved health (half the number of disabled years)
• Longer life (1.7 years of life per extra year of schooling)
For society
• Lower criminal justice costs
• Lower health-care costs
• Increased economic growth (Hanushek & Wößman, 2010)
o Present value to Sweden of raising PISA scores by 25 points: SEK 7 trillion
o Present value of ensuring all students score 400 on PISA: SEK 8 trillion
OECD (2009) Education at a Glance
OECD (2009) Education at a Glance
Estonia
Korea
Hungary
Slovak Republic
Poland
Slovenia
Czech Republic
France
Finland
Italy2
Portugal
Norway
OECD Average
United Kingdom
Israel
Spain
Sweden
Denmark
Turkey
Belgium
Netherlands
Canada
Switzerland
Austria1
30
Ireland1
40
United States
Greece
New Zealand
Impact of education on health
Percentage
100
90
80
70
60
50
Below upper secondary education
Upper secondary education
Tertiary education
20
10
0
Proportion of adults reporting good health, by level of education (OECD, 2010)
Which of the following categories of skill is
disappearing from the work-place most rapidly?
1.
2.
3.
4.
5.
Routine manual
Non-routine manual
Routine cognitive
Complex communication
Expert thinking/problem-solving
…but what is learned matters too…
Autor, Levy & Murnane, 2003
…now more than ever…
$35.00
$30.00
$25.00
Dropout
$20.00
HS Diploma
Some C ollege
BA/BSc
$15.00
Prof Degree
$10.00
$5.00
05
20
03
20
01
20
99
19
97
19
95
19
93
19
91
19
89
19
87
19
85
19
83
19
81
19
79
19
77
19
75
19
19
73
$0.00
Source: Economic Policy Institute
In fact low skill jobs are vanishing…
Over the last eight years, the UK economy has
shed 400 no-qualification jobs every day
Beyond Leitch (Patel et al., 2009)
…and recessions accelerate the trend…
Beyond Leitch (Patel et al., 2009)
There is only one 21st century skill
So the model that says learn while you’re at school, while you’re
young, the skills that you will apply during your lifetime is no longer
tenable. The skills that you can learn when you’re at school will not
be applicable. They will be obsolete by the time you get into the
workplace and need them, except for one skill. The one really
competitive skill is the skill of being able to learn. It is the skill of
being able not to give the right answer to questions about what you
were taught in school, but to make the right response to situations
that are outside the scope of what you were taught in school. We
need to produce people who know how to act when they’re faced
with situations for which they were not specifically prepared.
(Papert, 1998)
The test of successful education is not the amount of knowledge
that a pupil takes away from school, but his appetite to know
and his capacity to learn. If the school sends out children with
the desire for knowledge and some idea how to acquire it, it will
have done its work. Too many leave school with the appetite
killed and the mind loaded with undigested lumps of
information. The good schoolmaster is known by the number of
valuable subjects which he declines to teach.
The Future of Education (Livingstone, 1941 p. 28)
Where’s the solution?
 Structure
• Smaller/larger high schools
• K-8 schools/“All-through” schools
 Alignment
• Curriculum reform
• National strategies
 Governance
• Charter schools and private schools
• Specialist schools and academies
 Technology
• Computers
• Interactive white-boards
 Workforce reforms
• Classroom assistants
School effectiveness
 Three generations of school effectiveness
research
• Raw results approaches
o Different schools get different results
o Conclusion: Schools make a difference
• Demographic-based approaches
o Demographic factors account for most of the variation
o Conclusion: Schools don’t make a difference
• Value-added approaches
o School-level differences in value-added are relatively small
o Classroom-level differences in value-added are large
o Conclusion: An effective school is a school full of effective
classrooms
20
0
-60
-80
40
Iceland .
Finland .
Norway .
Sweden .
Poland .
Denmark .
Ireland .
Canada .
Spain .
New Zealand .
Australia .
United States .
Mexico .
Portugal .
Luxembourg .
Switzerland .
Greece .
Slovak Republic .
Korea .
Czech Republic .
Netherlands .
Austria .
Germany .
Italy .
Belgium .
Japan .
80
Hungary .
Turkey .
Sweden
100
Within schools
60
-20
-40
Between schools
Within schools
Betw een schools explained by social background of schools
Betw een schools explained by social background of students
Betw een schools not explained by social background
OECD PISA data from McGaw, 2008
Government schools
Government dependent private
Government independent private
%
Luxembourg
Japan
Italy
Switzerland
Finland
Denmark
Czech Republic
Sweden
Hungary
Austria
Portugal
United States
Netherlands
Slovak Republic
Korea
Ireland
Spain
Canada
Mexico
New Zealand
Germany
OECD average
United Kingdom
0
20 40 60 80 100-150
-100
-50
Private schools
perform better
0
50
100
Public schools
perform better
% of cohort reaching proficiency in 5 subjects
including English and Mathematics
CVA and raw results in England
1.0
0.8
0.6
0.4
0.2
960
1000
1040
School contextualized value-added (CVA) score
1080
Differences in CVA are often insignificant…
Middle 50%: differences
in CVA not significantly different
from average
(Wilson & Piebalga, 2008)
…are transient…
Future school effects for the 2014 cohort based on
2007 data with 95% confidence intervals
(Leckie & Goldstein, 2009)
…and are small
Scores of Swedish students in PISA
• 88% of the variability is within schools, so
• 12% of the variability is between schools, but
o One-third of this is explained by economic, social, and cultural status of
students, and
o One-sixth of this is explained by economic, social, and cultural status of
schools, so
• 6% of variability in student outcomes is attributable to schools
 So, if 15 students in a class reach proficiency in the
average school:
• 17 students will do so at a “good” school (1sd above mean)
• 13 students will do so at a “bad” school (1sd below mean)
It’s the classroom…
 In Sweden, variability at the classroom level is much
greater than at school level, so
• It doesn’t matter very much which school you go to
• But it matters very much which classrooms you are in…
It’s not class size
It’s not the between-class grouping strategy
It’s not the within-class grouping strategy
… and specifically, it’s the teacher…
Barber & Mourshed, 2007
Teachers make the difference
 The commodification of teachers has received widespread
support:
• From teacher unions (who understandably resist performancerelated pay)
• From politicians (who are happy that the focus is on teacher
supply, rather than teacher quality)
 But has resulted in the pursuit of policies with poor benefit
to cost
 To see how big the difference is, take a group of 50 teachers
• Students taught by the best teacher learn twice as fast as average
• Students taught by the worst teacher learn half as fast average
 And in the classrooms of the best teachers
• Students with behavioral difficulties learn as much as those
without
• Students from disadvantaged backgrounds do as well as those
from advantaged backgrounds
…so we have two choices…
 A classic labour force issue with 2 (non-exclusive)
solutions
• Replace existing teachers with better ones
• Improve the effectiveness of existing teachers
The ‘dark matter’ of teacher quality
 Teachers make a difference
 But what makes the difference in teachers?
Advanced content matter knowledge
Pedagogical content knowledge
Further professional qualifications (MA, NBPTS)
Total “explained” difference
<5%
10-15%
<5%
20-25%
Impact on achievement
 Raising the bar for entry into the profession so that
we no longer recruit the lowest performing 30% of
teachers would increases the proportion of students
passing an examination by less than 3%
 One student per class, in thirty years…
Or make the teachers we have better…
 Improve the effectiveness of existing teachers
• The “love the one you’re with” strategy
• It can be done
o Provided we focus rigorously on the things that matter
o Even when they’re hard to do
People like neuroscience
 Descriptions of 18 psychological phenomena
• Examples: mutual exclusivity, attentional blink
 Designed to be comprehensible without scientific training
 Each phenomenon was given four possible explanations
• Basic (without neuroscience)
o Good explanation (provided by the researchers)
o Bad explanation (e.g., circular reasoning)
• Enhanced (with neuroscience explanation)
o Good explanation
o Bad explanation
 Added neuroscience did not change the logic of the
explanation
 Participants randomly given one of the four explanations
 Asked to rate this on a 7-point scale (-3 to +3).
Sample explanations
Good explanation
Bad explanation
Without
neuroscience
The researchers claim that this ‘curse’
happens because subjects have trouble
switching their point of view to consider
what someone else might know,
mistakenly projecting their own
knowledge onto others.
The researchers claim that this ‘curse’
happens because subjects make more
mistakes when they have to judge the
knowledge of others. People are much
better at judging what they themselves
know.
With
neuroscience
Brain scans indicate that this ‘curse’
happens because of the frontal lobe brain
circuitry known to be involved in selfknowledge. Subjects have trouble
switching their point of view to consider
what someone else might know,
mistakenly projecting their own
knowledge onto others.
Brain scans indicate that this ‘curse’
happens because of the frontal lobe brain
circuitry known to be involved in selfknowledge. Subjects make more mistakes
when they have to judge the knowledge of
others. People are much better at judging
what they themselves know.
Seductive allure
Without neuroscience
With neuroscience
Explanation
Good
Bad
Good
Bad
Novices (n=81)
+0.9
–0.7
+0.9
+0.2
Students (n=22)
+0.1
–1.1
+0.7
+0.2
Experts (n=48)
+0.5
–1.1
–0.2
–0.8
(Weisberg et al., 2008)
Teachers do improve, but slowly…
Extra months per year of learning
0.5
0.4
0.3
0.2
0.1
Literacy
Numeracy
0
0
5
10
15
20
25
-0.1
-0.2
-0.3
-0.4
Y ears in service
Leigh, A. (2007). Estimating teacher effectiveness from two-year changes in student test
scores.
Pareto analysis
 Vilfredo Pareto (1848-1923)
• Economist, philosopher, etc., associated with
the 80:20 rule
 Pareto improvement
• A change that can make at least one person
(e.g., a student) better off without making
anyone else (e.g., a teacher) worse off.
 Pareto efficiency/Pareto optimality
• An allocation (e.g., of resources) is Pareto
efficient or Pareto optimal when there are no
more Pareto improvements
Schools are rarely Pareto optimal
 Examples of Pareto improvements
• Less time on marking to spend more time on planning
questions to use in lessons
• Increased use of peer assessment
• Larger classes with reduced teacher contact time
• Larger classes with increased teacher salaries
 Obstacles to Pareto improvements
• The political economy of reform
• In professional settings, it is incredibly hard to stop people
doing valuable things in order to give them time to do even
more valuable things
o e.g., “Are you saying what I am doing is no good?”
o e.g., “I care about my kids”.
Relevant studies








Fuchs & Fuchs (1986)
Natriello (1987)
Crooks (1988)
Bangert-Drowns, et al. (1991)
Kluger & DeNisi (1996)
Black & Wiliam (1998)
Nyquist (2003)
Dempster (1991, 1992)








Elshout-Mohr (1994)
Brookhart (2004)
Allal & Lopez (2005)
Köller (2005)
Brookhart (2007)
Wiliam (2007)
Hattie & Timperley (2007)
Shute (2008)
Cost/effect comparisons
Extra months of
learning per year
Cost/class-room/yr
Class-size reduction (by 30%)
4
£20k
Increase teacher content
knowledge from weak to strong
2
?
Formative assessment/
Assessment for learning
8
£2k
Intervention
Summary
 Securing future economic prosperity requires raising
achievement
 Raising achievement requires improving teacher
quality
 Improving teacher quality requires improving the
effectiveness of existing teachers
 Improving the effectiveness of existing teachers
requires professional development focused on
practice
 This requires new kinds of professional development