Transcript David Torgerson Phd

ARROW Trial Design
Professor Greg Brooks, Sheffield University, Ed Studies
Dr Jeremy Miles York University, Trials Unit
Carole Torgerson, York University, Ed Studies.
Professor David Torgerson York University, Trials Unit
Background



Information and communication
technology (ICT) is widely used in
schools.
There is little evidence that it is
beneficial.
A quasi experiment in Israel showed no
change in Hebrew literacy and a decline
in maths after implementation of ICT.
Background



To look at the effectiveness of ICT on literacy
a series of systematic reviews have been
undertaken with Dept of Ed Studies, York.
These have found no evidence of a benefit of
ICT on literacy.
A systematic review of trials looking at ICT
and spelling showed a modest, nonsignificant benefit on the learning of spelling.
Torgerson & Elbourne. JRR 2002;25:125.
Spelling Meta-analysis
Torgerson & Elbourne. JRR 2002;25:125.
Summary of evidence



Existing evidence of ICT and spelling
relies on 7 non-UK trials the largest of
which had only 99 pupils.
Evidence for benefit or harm is weak.
NEED to do more trials especially in the
UK.
ARROW trial

The ARROW study will evaluate a
computer based literacy package. A
computer will read text to pupils who
simultaneously read it on screen. The
pupil reads the text recording it on the
computer and finally they write the text
(either on the computer or on paper).
ARROW


The ARROW method has been used for
many years and pupils appear to make
progress with the method.
HOWEVER, existing data are only
before and after and its use is mainly
confined to children with low test scores
– thus inviting regression to the mean
effects.
Extending Use of ARROW

In the academic year 2004 it was
decided to offer ARROW to all Year 7
pupils in a large comprehensive school.
However, the school head and teachers
and developer of ARROW were
persuaded to undertake an RCT.
Method


All Year 7 pupils will be offered ARROW
either at the beginning of the first term
or at the end.
Pupils will be given a pre-test to assess
literacy levels and will be given
appropriate level of ARROW instruction.
Sample size

The total sample size is dictated by the
size of the year group. However, few
social science interventions improve
outcomes by more than 0.5 standard
deviations. Therefore we need at least
128 in our sample to detect this
difference. Our sample size will be 157.
Randomisation

To avoid subversion of randomisation the
children’s names have been sorted in
alphabetical order. A computer generated
randomisation list is then applied to these
names. Two copies of the allocation list are
held – an independent researcher visits the
school to check the children’s allocation
corresponds to the list of random numbers
(eg. The 10th child in the alphabet should
correspond to random number 10 on list).
Pre-test assessment

Pre intervention literacy levels are
assessed by independent researchers
BEFORE random allocation. This avoids
baseline testing being compromised by
children’s knowledge of their group
assignment.
Post test assessment



1 day after the intervention group have
received ARROW both groups will
receive post-tests.
12 weeks after ARROW all groups
receive second post-test.
Those giving and marking the post-tests
will be ‘blind’ to the group allocation of
the children.
Avoiding contamination

To avoid the control children obtaining
ARROW in the first term the programme
is delivered on laptop computers that
are removed after lessons.
Analysis


Main analysis is analysis of covariance,
comparing the two groups in terms of
their mean test scores at follow up.
Statistician will be blind to group
allocation (simply A or B).
Weaknesses of trial




A single school evaluation – the results may
not travel.
Results only applicable to the ARROW
software.
Torgerson & Elbourne meta-analysis found an
effect size of 0.35 to have good power, but to
demonstrate this would require > 200
participants, so trial is a little smaller than is
ideal.
Relatively short-term follow-up.
Trial Strengths


Trial is pragmatic – evaluating ‘real’
school practice rather than taking place
in a psychology lab. Children’s usual
teachers giving ARROW.
Trial is largest ever done in the field of
ICT and spelling, is UK based, and
using modern software and hardware.
Conclusion



ICT in schools has NEVER been
subjected to large robust RCTs.
ARROW is the first UK trial to evaluate
ICT and spelling.
Will inform school policy as to whether
to recommend continue using ARROW
or not.