Transcript Experiments in Education

Small Scale
Experiments in
Education
Dr Robert Coe
School of Education and Curriculum, Evaluation and Management (CEM) Centre
University of Durham
Headlines

Experiments are …




Necessary
Feasible
Ethical
Practitioners have …


An appetite for this kind of research
Not much time to do it
© 2004 Robert Coe
2
Examples of small scale RCTs




Underaspirers
Mentoring
Time of day
Feedback
© 2004 Robert Coe
3
4
Mentoring
Can academic mentoring improve
academic results?
© 2004 Robert Coe
Mentoring




In England, part of the KS3 Strategy
Backed by Government and private funding
‘Mentoring’ means a lot of different things
Research evidence is



Case studies
Feelings and perceptions of participants
Completely inadequate to infer impact
© 2004 Robert Coe
5
Neil Appleby’s Experiment




A randomised controlled trial involving 20
underachieving Y8 (12-13 year-old) students
Matched in pairs on ability and gender
Randomly allocated: one of each pair
mentored, the other not
Mentored group had 20 mins individually every
two weeks (11 sessions)



‘It nearly killed me’
Cost estimated at between £170 and £410 per
mentored pupil
Represents between 8-19% of the school’s annual
per pupil funding for the whole of their education
© 2004 Robert Coe
6
What the teachers said about the
mentored students …




7
“**** is a changed person this year she has
progressed greatly and is a superb helpful
student.”
“Better now, has achieved more, more
confident.”
“Generally a great improvement recently.”
“****’s attitude and effort have improved over
the year. He is a lot pleasanter and more willing
to participate in lessons particularly oral work,
he responds well to praise.”
© 2004 Robert Coe
What they said about the control
group …





“Has improved overall this term.”
“****’s attitude and effort have improved over
the last few months, she is now trying very
hard to achieve her target. Great effort.”
“Commended for attitude and progress.”
“**** has settled since the beginning of the
year.”
“**** has undergone quite a transformation
since September. Her attitude towards the
teacher and her learning have improved
drastically and she should be congratulated.”
© 2004 Robert Coe
8
9
Change in Teachers’ Ratings
of progress, effort and attitude
(English, maths and science combined)
group mean
+
group median
Mentored
Control
-6
-4
-2
0
2
4
Overall rating of change
© 2004 Robert Coe
6
8
10
What this proves



10
If you identify a group of underachieving pupils
at a particular time and then come back to
them after a few months, many of them will
have improved, whatever you did.
Others (the ‘hard cases’) will not have
improved, whether mentored or left alone.
The interpretation of this would have been very
different without a ‘control’ group
© 2004 Robert Coe
11
Academic achievement
Academic achievement in English
Difference between mentored and control groups
(NC levels)
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
-0.80
-1.00
-1.20
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = 0.03
© 2004 Robert Coe
--- Mentoring ---
12
Difference between mentored and control groups
(NC levels)
Academic achievement in Maths
1.00
0.80
0.60
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = -0.16
© 2004 Robert Coe
--- Mentoring ---
13
Difference between mentored and control groups
(NC levels)
Academic achievement in Science
1.00
0.80
0.60
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = 0.23
© 2004 Robert Coe
--- Mentoring ---
Ethical issues




14
University Ethics Advisory Committee
questioned the size of the study: ‘It seems a
rather small number to exclude a type 2 error? ‘
Withholding treatment from control group
Informed consent
Schools are not very ethical


Pupils never give consent for anything
Basic human rights (eg privacy) are violated all the
time
© 2004 Robert Coe
15
Time of day
Do children learn better in the morning
or afternoon?
© 2004 Robert Coe
16
Time of day effects on learning


Do children learn better in the morning or
afternoon?
Series of experiments by Val Dowson (MA
practicum)




Y2-Y6
comprehension, maths, non-verbal reasoning
Effect sizes 0.84 to –0.16
Overall, the afternoon seems better
© 2004 Robert Coe
1.46
0.80
0.14
-1.0
© 2004 Robert Coe
0.84
0.18
0.83
0.15
-0.52
1.0
-0.52
1.03
0.83
0.41
-0.21
0.16
0.0
1.50
0.92
0.78
0.30
0.09
-0.61
-0.85
0.54
0.80
0.18
0.04
-0.44
-0.58
0.66
0.66
0.16
-0.46
0.04
-0.32
-0.16
-0.58
OVERALL AVERAGE
(IMMEDIATE)
Non-verbal reasoning,
yr 3, Immediate
Maths, yr 3, 4 weeks
Maths, yr 3, 1 week
Maths, yr 3, Immediate
Comprehension/recall,
yr 6, 1 week
Comprehension/recall,
yr 6, Immediate
Comprehension/recall,
yr 2, 1 week
Comprehension/recall,
yr 2, Immediate
Comprehension/recall,
yr 2, 1 week
Comprehension/recall,
yr 2, Immediate
Comprehension/recall,
yr 3, 1 week
Comprehension/recall,
yr 3, Immediate
Standardised Effect Size (+ve indicates afternoon better)
Time of day effects: results
17
2.0
Effect Size estimate
Upper confidence limit
Lower confidence limit
0.78
0.54
0.28
0.01
18
Feedback
Does giving performance feedback to
teachers help their students?
© 2004 Robert Coe
Design





Cluster RCT with pre- and post-test
9 schools and colleges teaching A level
Whole departments allocated to receive
feedback or not
Confidential feedback to individual teachers on
past students’ value-added and current
student’s predictions
Control group got feedback at the end
© 2004 Robert Coe
19
20
Pre-test equivalence
treatment
control
-10
-5
0
Residuals 1994-6
© 2004 Robert Coe
5
10
21
Post-test difference
treatment
control
-10
-5
0
Residuals 1997
© 2004 Robert Coe
5
10
Effect on students’ A level grades
22
Estimates for the effects of feedback (with 95% confidence intervals)
From 5 multilevel models
1.0
0.8
A level grades
0.6
0.4
0.2
0.0
-0.2
-0.4
-0.6
Model 1: unadjusted


Model 2: adj. for
GCSE
Model 3: adj for
GCSE, PO
Model 4: adj for
GCSE, PO, sex
Model 5: adj for
GCSE, PO, sex &
prev resid
Students whose teachers received the feedback achieved about
a third of an A level grade higher
Intervention cost virtually nothing
© 2004 Robert Coe
23
Analysis at the teacher level
Mean Standardised Residual
1.50
1.00
.50
1997
control
feedback
.00
-.50
-1.00
-1.00
© 2004 Robert Coe
-.50
.00
1994-6
.50
1.00
1.50
24
Underaspirers
How can we help pupils with low
aspirations?
© 2004 Robert Coe
Identifying ‘under-aspirers’





© 2004 Robert Coe
Yellis Project allows schools to monitor
progress and attitudes.
Year 10 students complete a test of developed
abilities and a questionnaire.
Able students who say they are not planning
to stay in education are identified as
‘underaspirers’.
Schools then ‘mentor’, ‘target’ etc.
In 1999 some schools were asked if they
would mind getting only half the list (selected
at random).
25
Results



120 year 10 students in 15 ‘typical’
schools
Half ‘named’, half ‘not named’
In terms of achievement in GCSEs (value
added):



© 2004 Robert Coe
Named students did worse in 12 of 15
schools
Average difference 0.3 grades per subject
Overall effect size of naming: -0.38
26
27
Named
Not named
-4
-3
-2
-1
0
Average Residual / value added (GCSE grades)
© 2004 Robert Coe
1
2
28
Was more counselling better?
1.5
1.0
Average Residual
.5
0.0
-.5
-1.0
Named?
-1.5
Y
-2.0
-2.5
N
0
10
No of times counselled
© 2004 Robert Coe
20
30
196 students in29
26 schools
Underaspirers (2)
GCSEs in 2002
group mean
+
group median
Gaussian confidence interval for the
difference between the two groups
Named
Not named
-6
-5
-4
-3
-2
-1
0
Average Residual
© 2004 Robert Coe
ES = 0.12
1
2
Issues




Dramatic negative result, but not replicated
No informed consent from students
Not much known about what the intervention
actually was
Experiment cost virtually nothing to do
© 2004 Robert Coe
30
31
Making evidence-based decisions



The problem (and possible solutions)
must be ‘generalisable’
Agreement about outcomes?
What evidence exists already?





© 2004 Robert Coe
Theory (formal and informal)
Experience
Research
Systematic reviews of research
Conduct an experiment
Why involve teachers?







Only those who do the job can ask the right
questions in the right ways
Only those close to the outcomes can provide a
rich and detailed understanding of them
Many teachers are already experimenting
Assimilation, not dissemination: other people’s
ideas don’t have the same impact
We must evaluate actual implementation, not just
ideal policy
Only multi-site trials can give generalisable
findings
The process of doing the research is valuable
© 2004 Robert Coe
32
Dr Robert Coe
Curriculum, Evaluation and Management (CEM) Centre
University of Durham
Mountjoy Research Centre 4
Stockton Road
Durham DH1 3UZ
Tel: (+44/0)191 33 44 184
Email: [email protected]
http://www.cemcentre.org
Fax: (+44/0)191 33 44 180