Transcript Academic mentoring in schools: a small RCT to evaluate a

Academic mentoring in schools: a
small RCT to evaluate a large policy
Randomised Controlled trials in the Social
Sciences: Challenges and Prospects
University of York, 13-15 Sept 2006
Robert Coe
Neil Appleby
Mentoring
• In England, part of the KS3 Strategy
• ‘Learning Mentors’ part of Excellence in
Cities
• Now part of ‘Every Child Matters’
• ‘Mentoring’ means a lot of different things
– Group / individual; location; duration; goals
(Sipe and Roder, 1999)
• Academic mentoring in schools
– Takes place in school
– Aims to improve academic achievement
2
Evidence of effectiveness
• Meta-analysis by DuBois et al (2002)
– Modest positive overall effect (SMD=0.14)
– “evidence of an overall favorable effect of mentoring is notably
lacking under circumstances in which participating youth have
been identified as being at risk solely on the basis of individuallevel characteristics (e.g. academic failure).”
• UK review by Hall (2003)
– “There is a very poor evidence base in the UK. Claims are made
for the impact of mentoring but there is as yet little evidence to
substantiate them.” (p15)
– “Mentoring is in danger of being unsuccessful if any of the
following conditions apply:
• social distance and mismatch between the values of mentor and
mentee
• inexpert or untrained mentors
• mismatch between the aims of the mentoring scheme and the
needs of the person being mentored
• conflict of roles such that it is not clear whether the mentor is to act
on behalf of the person being mentored or of ‘authority’.” (p24)
3
Justification for the current
policy of Learning Mentors
“Since their introduction in 1999, learning mentors have
made a significant contribution to re-engaging pupils in
learning and helping to raise standards in primary and
secondary education. For example, research has shown
that pupils receiving support from learning mentors were
one and a half times more likely to achieve five or more
GCSEs at grades A* to C than young people with similar
prior attainment who had not been mentored.1 In many
cases learning mentors succeeded in raising pupil
performance above the levels indicated by these pupils’
Key Stage 3 outcomes.”
Supporting the new agenda for children’s services and schools: the
role of learning mentors and co-ordinators. DfES, 2005,
4
To be worth considering, a study
would have to
• Be conducted in the UK
• Include a comparison group
• Provide evidence of initial equivalence and
the reasons for membership of
mentoring/comparison groups
• Describe the ‘mentoring’ intervention
adequately
• Include an outcome measure of academic
achievement
5
Neil Appleby’s Experiment
• A randomised controlled trial involving 20
underachieving Y8 (12-13 year-old) students
• Matched in pairs on ability and gender
• Randomly allocated: one of each pair
mentored, the other not
• Mentored group had 20 mins individually every
two weeks (11 sessions)
– ‘It nearly killed me’
– Cost estimated at between £170 and £410 per
mentored pupil
– Represents between 8-19% of the school’s annual
per pupil funding for the whole of their education
6
What the teachers said about
the mentored students …
• “**** is a changed person this year she has
progressed greatly and is a superb helpful
student.”
• “Better now, has achieved more, more
confident.”
• “Generally a great improvement recently.”
• “****’s attitude and effort have improved over
the year. He is a lot pleasanter and more willing
to participate in lessons particularly oral work,
he responds well to praise.”
7
What they said about the
control group …
• “Has improved overall this term.”
• “****’s attitude and effort have improved over
the last few months, she is now trying very hard
to achieve her target. Great effort.”
• “Commended for attitude and progress.”
• “**** has settled since the beginning of the
year.”
• “**** has undergone quite a transformation
since September. Her attitude towards the
teacher and her learning have improved
drastically and she should
be
congratulated.”
8
Change in Teachers’ Ratings
of progress, effort and attitude
(English, maths and science combined)
group mean
+
group median
Mentored
Control
-6
-4
-2
0
2
4
Overall rating of change
9
6
8
10
What this proves
• If you identify a group of underachieving
pupils at a particular time and then come
back to them after a few months, many of
them will have improved, whatever you
did.
• Others (the ‘hard cases’) will not have
improved, whether mentored or left alone.
• The interpretation of this would have been
very different without a ‘control’ group
10
Academic
achievement
Academic achievement in English
Difference between mentored and control groups
(NC levels)
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
-0.80
-1.00
-1.20
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = 0.03
11
--- Mentoring ---
Difference between mentored and control groups
(NC levels)
Academic achievement in Maths
1.00
0.80
0.60
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = -0.16
12
--- Mentoring ---
Difference between mentored and control groups
(NC levels)
Academic achievement in Science
1.00
0.80
0.60
0.40
0.20
0.00
Dec 01
May 02
Dec 02
May 03
-0.20
-0.40
-0.60
(+ve scores indicate mentored better than control
1 NC level = two years)
Effect size = 0.23
13
--- Mentoring ---
Ethical issues
• University Ethics Advisory Committee
questioned the size of the study: ‘It seems
a rather small number to exclude a type 2
error? ‘
• Withholding treatment from control group
• Informed consent
• Schools are not very ethical
– Pupils never give consent for anything
– Basic human rights (eg privacy) are violated
all the time
14
Challenges and prospects
• Limitations of ‘evaluation by participants’
• Importance of control group
• Why does education suffer from these
fashions?
• You can do RCTs in education
15
Dr Robert Coe
Curriculum, Evaluation and Management (CEM) Centre
University of Durham
Mountjoy Research Centre 4
Stockton Road
Durham DH1 3UZ
Tel: (+44/0)191 33 44 184
Fax: (+44/0)191 33 44 180
Email: [email protected]
www.cemcentre.org
16