Transcript Uses of Value Added

Maintaining Momentum in Primary School:
messages from research and evaluation
Presentation Prepared for the Social Mobility and Life
Chances Forum Maintaining Momentum, Milton Hill Oxford
December 3-4 2004
Pam Sammons & Kathy Sylva
University of Nottingham/University of Oxford
Content of Presentation

What is the impact of multiple disadvantage and does
pre-school promote better child outcomes at primary
school? - Evidence from the EPPE study

 Does the primary school a child attends ‘make a
difference’ to their educational outcomes? - Evidence
from school effectiveness research
 What is the evidence of ‘improvement through
inspection’ and has this benefited disadvantaged
groups? - Evidence from an evaluation of the impact
of Ofsted
 What kinds of specific interventions promote better
outcomes for disadvantaged groups of children? –
Evidence from the SPOKES study
Attainments of EPPE Sample at end of Year 1 by Multiple Disadvantage
Multiple
Disadvantage
Index
Primary Reading
standardised score
N
mean
sd
0
588
104.1
13.7
587
106.7
14.3
1
679
102.7
14.2
680
102.7
15.0
2
532
98.8
15.0
531
99.8
14.9
3
336
96.2
13.7
333
96.1
14.7
4
221
92.1
15.0
219
90.0
13.6
5 plus
167
89.9
13.9
165
89.9
14.6
All pupils
2532
99.6
15.0
Maths 6
standardised score
N mean
sd
2515 100.2 15.6
Impact of quality and duration
Pre-reading at school entry
0.7
0.6
Effect size
0.5
0.4
0.3
0.2
0.1
0
Low duration
Low quality
High duration
High quality
Effect of pre-school (v. no pre-school) on socialbehavioural outcomes at school entry
0.6
0.5
Effect size
0.4
0.3
0.2
0.1
0
Cooperation and
conformity
Independence and
concentration
Peer sociability
The contribution of social class and pre-school to
literacy attainment (age 7)
WRITING at key stage 1, social class and
pre-school experience
2.8
2.6
2.6
2.4
Pre-school
2.4
2.2
No pre-school
2.0
Mean year 2 writing level
Mean year 2 reading level
READING at key stage 1, social class and
pre-school experience
Pre-school
2.2
2.0
Expected minimum
1.8
Expected minimum
No pre-school
1.8
Professional
Skilled
Un/semi skilled
1.6
Professional
Skilled
Un/semi skilled
Social class by occupation
Social class by occupation
What reduces the risk of SEN?
● Higher quality & longer duration of pre-school
● Integrated settings and Nursery schools
● Good home learning environment and employed
parent(s)
Pre-school reduced proportion of children ‘at
risk of SEN from 1:3 to 1:5
Focus of SER
 The central focus a belief in the potency of
social institutions
‘the idea that schools matter, that schools do have major
effects upon children’s development and that, to put it
simply, schools do make a difference’ (Reynolds & Creemers,
1990)
‘Effectiveness is not a neutral term. Defining the
effectiveness of a particular school always requires
choices among competing values … the criteria of
effectiveness will be the subject of political debate’
(Firestone, 1990)
Aims & Goals of Early SER
to promote Equity and Excellence
 Clientele - poor/ethnic minority students
 Subject matter - basic skills reading &
maths
 Equity - children of urban poor should
achieve at same level as those of
middle classes
Focus on Student Outcomes
‘For us the ‘touchstone’ criteria to be applied to all
educational matters concern whether children
learn more or less because of the policy or
practice’ Reynolds 1997
‘An effective school is one in which students
progress further than might be expected from
consideration of its intake’
Mortimore 1991
SER seeks to identify the ‘Value Added’ by
schools to student outcomes
Methodology
 mainly quantitative, but case studies
important
 values reliability and replicability
 seeks to make generalisations
 works in partnership with
practitioners
 values the views and perceptions of
teachers, students and parents
The Impact of Intake
‘ ‘Natural justice demands that schools are
held accountable only for those things
they can influence (for good or ill) and not
for all the existing differences between
their intakes’ (Nuttall 1990)
SER seeks to disentangle the impact of
prior attainment and background
characteristics from the impact of school
and classes/teachers on students’
progress/social or affective outcomes
Example of More Effective or Less Effective
School Profiles at Key Stage 1
Progress
Effectiveness English
Maths Science
Category
positive*
X
positive
X
X
As expected
negative
O
negative*
O
O
*value added statistical outlier, p<0.05
X more effective profile
O less effective profile
Example of value added feedback from Improving
School Effectiveness Project: Primary Schools’ AAP
Results
Value added effectiveness
category
Positive Outlier (p<0.05) *
AAP
Mathematics
n
%
AAP Reading
n
%
10
23
5
11
Positive (non-significant)
As expected
7
16
17
39
Negative (non-significant)
As expected
15
35
19
43
Negative Outlier (P<0.05) *
11
26
3
7
N of primary schools= 44 * p<0.05 , MacBeath & Mortimore, 2000
Defining Consistency
Within school comparisons focus on
internal variation in effects
 For different cognitive & non-cognitive
outcomes
 By different year groups within each school,
including variations in class or teacher
effects
 For different pupil groups
- boys/girls
- initial low/high attainers,
- low SES/high SES
Effectiveness is a relative concept
which is time and outcome specific
 Effective in promoting which outcomes?
the what of effectiveness
 Effective for which student groups?
the who of effectiveness
 Effective over what time period?
the when of effectiveness
Differential Effectiveness
 The size of school effects for black students were almost
twice as large as for white students in the US (Coleman et al
1966)
 Differences between public and private schools almost
twice as large for low SES students as for middle class
students, differences between schools for high SES
students small in US (Bryk & Raudenbush, 1992)
 School effects vary for students by race and low prior
attainment in England. School effects larger for initially
low attaining and for black Caribbean students (Nuttall et al
1989)
 Primary school effects vary for students with low
compared with high initial attainment in England, being
larger for low initial attainers (Sammons et al 1993)
Equity Implications
 Dutch primary schools are highly stable in
effectiveness across grades for low SES students,
less stable in effectiveness across grades for high
SES students (Bosker 1995)
‘Schools matter most for underprivileged
and/or initially low achieving students.
Effective or ineffective schools are especially
effective or ineffective for these students’
After Scheerens & Bosker 1997
The Processes of Effective Schools
1. The processes of effective leadership
2. The processes of effective teaching
3. Developing & maintaining a pervasive
focus on learning
4. Producing a positive school culture
5. Creating high (& appropriate expectations
for all)
6. Emphasising student responsibilities &
rights
7. Monitoring progress at all levels
8. Developing staff skills at the school site
9. Involving parents in productive
& appropriate ways
After Teddlie & Reynolds 2000
The ‘ineffective’ school
(Reynolds 1995)
 Non-rational approach
to evidence
 fear of outsiders
 dread of change
 capacity for blaming
external conditions
 set of internal cliques
 lack of competencies for
improvement
..may have inside itself
multiple schools
formed around cliques
and friendship groups
.. There will be none of
the organisational,
social, cultural and
symbolic tightness of
the effective school
Empirical Confirmation of SE : MetaAnalyses




cooperation
school climate
monitoring at school and class level
opportunity to learn (content coverage - homework time)
 parental involvement
 pressure to achieve
 school leadership After Scheerens & Bosker 1997
The most powerful factors are located at the
classroom level. Schools should address ‘proximal
variables’ like curriculum, instruction and
assessment which emphasis student outcomes
Wang et al 1993
Processes for School Improvement










Clear leadership
Developing a shared vision & goals
Staff development & teacher learning
Involving pupils, parents & community
Using an evolutionary development planning process
Redefining structures, frameworks, roles
& responsibilities
Emphasis on teaching & learning
Monitoring, problem-solving & evaluation
Celebration of success
External support, networking & partnership
Significance of School Effects
Although the differences in scholastic attainment
achieved by the same student in contrasting
schools is unlikely to be great, in many instances it
represents the difference between success and
failure and operates as a facilitating or inhibiting
factor in higher education.
 When coupled with the promotion of other
pro-social attitudes and behaviours, and the
inculcation of a positive self-image,the
potential of the school to improve the life
chances of students is considerable.
Mortimore 1998:143
Impact of Inspection: Outcomes of
special measures over 10 years
Primary
N
Special
% N
Removed 799 89.6 114
%
Secondary PRUs
Total
N
N
%
77.0 167 76.6
N
%
%
18 60.0 1098 85.3
from
special
measures
Closed
93
10.4 34
Total
892
148
23.0
51
218
23.4 12 40.0 190 14.6
30
1288
Perceptions of benefits of inspections: 2002/03;
comparison of head teachers’ and teachers’ views
Benefits outweigh Benefits and negative Negative effects
negative effects
effects equally
outweigh benefits
%
balanced %
%
60
25
14
45
32
22
Head
Teachers
N = 2801
Teachers
N = 2436
Judgements of extent of Improvement of primary
and secondary schools since their last inspection
(2002/03 Annual Report)
Primary schools
23
Secondary schools
24
0%
Excellent/
very good
44
43
20%
Good
40%
Satisfactory
60%
26
8
25
8
80%
Unsatisfactory/
poor
100%
Primary schools: change of inspection judgements
from first to second inspection (percentage of
schools)
Leadership and management
24
32
9
Teaching
0
%
Significant
improvement
Improvement
27
35
20
%
No
change
13
42
40
%
60
%
Deterioration
12
80
%
Significant
deterioration
4
2
100
%
Percentage of 11 year-old pupils reaching level 4
and above in English, mathematics and science
100
Government target for English and mathematics for 2006
78
Percentage of pupils
80
69
62
60
57
63 62
69
65
71 69
87
85
75
72
75
71
87
86
75 73
75 73
59
54
40
20
0
1996
1997
English
1998
1999
Maths
2000
2001
2002
Science
2003
The proportion of good or better teaching in primary
schools
Percentage of schools
100
80
73
74
76
75
1999/00
2000/01
2001/02
2002/03
62
55
60
45
40
20
0
1996/97
1997/98
1998/99
300
Source: PIRLS 2001 International Report: IEA’s Study of Reading Literacy Achievement in Primary Schools
Belize
Morocco
Kuwait
Iran, Islamic Rep of
Argentina
Colombia
Macedonia, Rep of
Turkey
Moldova, Rep of
Cyprus
Norway
International Avg.
Slovenia
Israel
Romania
Iceland
Slovak Republic
Greece
France
Hong Kong SAR
Russian Federation
Singapore
Scotland
New Zealand
Czech Republic
Germany
Italy
United States
Hungary
Lithuania
Canada (Ontario,Quebec)
Latvia
Bulgaria
England
Netherlands
Sweden
International Comparisons of Reading Attainment
2001:IEA
Distribution of Reading Achievement in 9-10 year olds in 2001
575
550
525
500
475
450
425
400
375
350
325
Is improvement greater in schools
facing challenging circumstances?
Progress in closing the socio-economic attainment gap
in primary schools 1996-2001
MEDIAN SCHOOL ACHIEVEMENT IN TERMS OF PROPORTION REACHING EXPECT ED LEVEL,
GROUPED BY ELIGBILITY FOR FREE SCHOOL MEALS
Key Stage 1
Reading
FSM band
8% or less
8+ to 20%
20+ to 35%
35+ to 50%
Above 50%
Total
1996
90
83
75
67
63
82
SOURCE: DfES School level data
2001
92
87
81
75
71
87
Key Stage 2
English
1996
74
64
51
41
34
60
2001
87
78
69
61
57
78
Percentage of unsatisfactory /poor lessons in
primary schools going into special measures and
two years after coming out (2002/03)
Percentage of lessons
unsatisfactory or poor
80
70
60
50
40
30
20
10
0
-10
Lessons from S10 inspections when schools went into special measures
Lessons from S10 inspections two years after coming out of special measures
Disadvantaged pupils are over-represented in schools judged to
require special measures
Percentage (%)
Percentage of prim ary schools entering SM w ithin their
FSM bands com pared against National distribution- 2002/03
50
40
30
20
10
0
1
2
3
4
5
Free School Meals (FSM) Bands
% placed in Special Measures
National % in each FSM Band
Improving City Schools: key features of
teaching
 a high degree of consistency across the school
 high expectations of pupils, matched by well planned
support to help them meet the challenges of the work
 skilful management of pupils in classrooms and
effective use of time and resources
 motivating teaching methods & materials, planned
with the improvement of basic skills in mind
Ofsted 2000
Challenges for 21st century
Pupils from disadvantaged backgrounds are more
likely than others to experience educational failure
Reasons for eradicating school failure
 philosophical/ethical - to promote fairness improvement in
quality of life and opportunities for all groups, to encourage
positive attitudes to future learning and self-esteem
 political - to promote social cohesion and inclusion and
empower young people as citizens to participate in a
successful democracy
 economic - to promote future prosperity & prevent waste of
talent & avoid social/economic burden on Governments
Maintaining Momentum in the Primary Phase :
messages from research & evaluation
 Pre-school provides children with a better start to school and is
particularly important in improving attainment for low SES pupils,
the impact is still evident at age 7 years
 Schools vary in their effectiveness. For disadvantaged groups
the effectiveness of the primary school attended is particularly
important.
 SER provides an important evidence-base on the correlates of
effective schools and teachers and has stimulated school
improvement initiatives at national and local level.
 Inspection, has helped raise overall attainment levels and
improved the quality of teaching in primary schools.
 Inspection has acted as a powerful catalyst for improvement of
weaker schools and this has benefitted disadvantaged pupil
groups because they are over represented in such schools.
 For the most vulnerable groups of pupils intensive, structured
and targetted interventions are needed at an early stage.
The EPPE team
Kathy Sylva University of Oxford
Edward Melhuish Birkbeck, University of London
Pam Sammons University of Nottingham
Iram Siraj-Blatchford Institute of Education, University of
London
Brenda Taggart Institute of Education, University of London
http://www.ioe.ac.uk/projects/eppe
http://www.dfes.gov.uk/research/
EPPE is an ESRC TLRP (Affiliate) project