Transcript Research & Development

Perspectives on the
Future of Assessment
in England and Internationally
Robert Coe
CEM conference, 25th January 2012
1
Outline
 Background to CEM
 Current context of assessment
 Key questions about the future of
assessment:
o
o
o
o
2
Should we be using standardised tests?
How can classroom assessment support
learning?
Can assessment data identify good teachers?
How can monitoring and feedback of
performance support improvement?
CEM’s Aims
To help educators improve learning and
other educational outcomes, through
o
o
o
o
3
Assessments that support learning
Monitoring and feedback systems for selfevaluation
Rigorous evaluation of the impact of different
approaches
Promotion of evidence-based practices and
policies
CEM’s Achievements
 Providing monitoring systems for schools
for almost 30 years – we led the world
from the North East of England
 CEM assessments are used by
o
o
o
1.1 million students each year
More than 50% of UK secondary schools
Schools in over 40 countries
 Largest provider of computerised adaptive
tests outside US
 The largest educational research unit in a
UK university
4
Current context of
assessment
5
6
GCSE & A level …
 Existing qualifications in England are the
legacy of an out-dated, amateurish view of
3/
assessment
10 Could do better
 Good: high-stakes assessments are based
on what has been studied
 Bad: examinations often trivialise the range of
skills, knowledge and understanding that
have been (should be?) taught, are poorly
conceived and constructed and validity is an
afterthought (at best).
7
Low level skills are easier to
assess
• Bloom’s Taxonomy • SOLO Taxonomy
– Knowledge
– Comprehension
– Application
– Analysis
– Synthesis
– Evaluation
8
– Pre-structural
– Uni-structural
– Multi-structural
– Relational
– Extended abstract
NC Review








9
NC is just a part of curriculum
Broaden at KS4
2-year Key Stages
Clarify relationship between PoS &
assessment
Ensure all students are ‘ready to progress’
‘High expectations for all’
Detailed profiles, not general levels
No change to GCSE
Michael Gove on A level
standards
“Researchers at Durham University have been particularly
good at challenging the growth in grade performance. One
piece of analysis from Durham concluded that between 1996
and 2007, the average grade achieved by GCSE candidates
of the same ‘general ability’ rose by almost two thirds of a
grade. And the rise, they argued, is particularly striking in
some subjects: in 2007, pupils received a full grade higher in
maths, and almost a grade higher in history and French, than
pupils of the same ability when they sat the exams in 1996.
Similar trends have been found at A level. Academics at
Durham found that in 2007, A level candidates received
results that were over two grades higher than pupils of
comparable ability in 1988. And pupils who would have
received a U in Maths A-Level – that’s a fail – in 1988 received
a B or C in 2007.”
Ofqual Standards Summit, 13 Oct 2011
10
Rising standards
11
Grade slippage at A level
Average grade achieved by students with the same ability
(ITDA=50)
Curriculum 2000 and new TDA
A level grade
C
----------------------
Biology
English (Lit)
French
Geography
History
Maths
Weighted avg of 40 subjs
B
D
-----------
E
12
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
1995
1994
1993
1992
1991
1990
1989
1988
U
International surveys
Performance of England in international surveys
560
Reading (age 11, PIRLS)
550
Standardised test score
540
Science (age 14, TIMSS)
530
Science (age 10, TIMSS)
520
25 point rise in PISA
Scientific literacy (age 15, PISA)
=
510
+£4,000,000,000,000 GDP
Maths (age 14, TIMSS)
500
Reading literacy (age 15, PISA)
Mathematical literacy (age 15,
PISA)
490
Maths (age 10, TIMSS)
Year
13
2010
2009
2008
2007
2006
2005
2004
2003
2002
2001
2000
1999
1998
1997
1996
1995
1994
480
Relative severity (corrected tariff)
FilmStud
ArtPhoto
MediaStud
ArtText
ArtGraph
TravelTour
Sociology
BusAppl
FineArt
ArtDes
Drama
EngLangLit
HealSocCare
EconBus
BusStud
EngLang
Law
DTProdDes
RS
EngLit
GovPol
ICTAppl
Geography
Psychology
ClassCiv
PESportStud
ICT
History
AccouFin
MusicTech
Economics
Maths
LogicPhil
Music
Computing
French
MathFur
BiolHuman
Biology
Chemistry
Physics
A fair UCAS points tariff?
Leniently graded
120
14
60
20
0
Severely graded
180
160
A*
140
A
100
B
80
C
D
40
E
Key questions about the
future of assessment
15
Should we be using
standardised tests?
16
Cons
Pros
• Test only a limited part of
what can be learnt
• Methods (eg multiple
choice) are limited and
constraining
• Focus on short, closed
tasks
• Emphasise (fixed) ability
• Well designed tests can
cover the full range of
content and methods
• Standardisation gives
valuable reference point
for performance
• Teacher-created
assessment is generally
expensive, hard to
standardise, unreliable
and biased
Formative Assessment (AfL)
 1998: Black & Wiliam’s review – strong
evidence of power of FA
 Support from governments to implement
 Are teachers actually doing it?
 Do we know what ‘it’ is?
 Have there been improvements in
learning?
 How do you get a teacher is not currently
doing it faithfully to do so?
17
What happens if you get a good
teacher for several years?
18
Tymms et al (2009)
Long-term effects of a good
primary teacher
 Having a good teacher (+1SD in VA, ie top
16%) in a single year
o
o
o
Raises test scores that year by 0.1 SD; about
1/3 of the gain is sustained
Raises earnings by about 1% at age 28
Is worth paying $4,600 per child to retain
 Replacing a very poor (bottom 5%)
teacher with an average teacher is worth
$267k to each class they teach
Chetty et al (2011)
19
www.suttontrust.com
20
Overview of value for money
Promising
10
May be
worth it
Effect Size (months gain)
Feedback
Meta-cognitive
Pre-school
Peer tutoring
1-1 tutoring
Homework
0
£0
Summer
schools
Parental
AfL
involvement
Learning Individualised
Sports
learning
styles
Arts
Performance
Ability grouping
pay
Cost per pupil
21
ICT
Smaller
classes
After
school
Not
worth it
£1000
Teaching
assistants
Is that it?
Have we solved
the problem of how
to improve
attainment?
22
Implementation
 These strategies have been shown to be costeffective in research studies
 But when we have tried to implement evidencebased strategies we have not seen system-wide
improvement
 We don’t know how to get schools/teachers who are
not currently doing them to do so in ways that are
o
o
o
o
23
True to the key principles
Feasible in real classrooms – with all their constraints
Scalable & replicable
Sustainable
CEM’s Aims
To help educators improve learning and
other educational outcomes, through
o
o
o
o
24
Assessments that support learning
Monitoring and feedback systems for selfevaluation
Rigorous evaluation of the impact of different
approaches
Promotion of evidence-based practices and
policies