Transcript Carnegie Mellon Micro-analysis of Fluency Gains in a Reading Tutor that Listens: Jack Mostow and Joseph Beck Project LISTEN (www.cs.cmu.edu/~listen) Carnegie Mellon University Society for the Scientific.

Carnegie
Mellon
Micro-analysis of Fluency Gains in a
Reading Tutor that Listens:
Jack Mostow and Joseph Beck
Project LISTEN (www.cs.cmu.edu/~listen)
Carnegie Mellon University
Society for the Scientific Study of Reading
12th Annual Meeting, June, 2005
Funding: National Science Foundation, Heinz Endowments
Mostow & Beck,
Project LISTEN
1
11/7/2015
Carnegie
Mellon
Wide vs. repeated guided oral reading
Guided oral reading builds fluency [NRP 00].
 Typically repeated oral reading
Is repeated reading better than wide reading?
 Unclear! [Kuhn & Stahl JEP 03]
Past work analyzed reading rates on passages and word lists.
 E.g. scramble word order to study context effects [Levy, …]
This talk: use finer-grained Reading Tutor data.
Mostow & Beck,
Project LISTEN
2
11/7/2015
Carnegie
Mellon
Project LISTEN’s Reading Tutor:
Rich source of guided oral reading data
2002-2003 database:





Mostow & Beck,
Project LISTEN
3
8 diverse schools
600 students (K-6)
26,000 sessions
600,000 sentences read
4 million words heard
11/7/2015
Carnegie
Mellon
Reading speeds up with practice: example
Initial encounter of muttered:
 I’ll have to mop up all this (5630 ms) muttered Dennis to
himself but how
5 weeks later (different word pair in different sentence):
 Dennis (110 ms) muttered oh I forgot to ask him for the money
Word reading time = latency + production time  1/fluency
 Beck et al. [TICL 04] used latency to assess proficiency (R2>.8).
What predicts word reading time?
Mostow & Beck,
Project LISTEN
4
11/7/2015
Carnegie
Mellon
Proficient readers are faster.
800
R2 = 0.9663
word reading time (ms)
750
700
650
600
550
500
450
400
0
Mostow & Beck,
Project LISTEN
1
2
3
4
WRMT pretest score (grade)
5
5
6
11/7/2015
Carnegie
Mellon
Long words are slower.
1000
2
R = 0.9831
word reading time (ms)
900
800
700
600
500
400
1
Mostow & Beck,
Project LISTEN
2
3
4
5
6
7
8
9
10
11
word length (letters)
6
11/7/2015
Carnegie
Mellon
Reading time speeds up over successive
encounters, but by less and less.
word reading time (ms)
680
R2 = 0.9931
660
640
620
600
580
560
540
520
1
2
3
4
5
6
7
# previous encounters
Mostow & Beck,
Project LISTEN
7
11/7/2015
Carnegie
Mellon
Linear model of word reading speedup
Predicted variable is speedup on the same word
 Reduction in word reading time
 From one (“practice”) encounter of a word
 To the next (“test”) encounter
N = 243,172 opportunities for speedup




By 352 students (gr 1-6) with WRMT pretest scores
Include rereading as practice but not as test
Exclude encounters after the first 8
Exclude 36 stop words (the, a, …)
Mostow & Beck,
Project LISTEN
8
11/7/2015
Carnegie
Mellon
Predictors of speedup
What is the student’s reading level?
How many letters long is the word?
How often has the student seen the word before?
Mostow & Beck,
Project LISTEN
9
11/7/2015
Carnegie
Mellon
Contextual predictors
Has the student seen the practice sentence before?
(Students have not seen test sentences before.)
Has the student seen the practice word pair before?
Has the student seen the test word pair before?
Mostow & Beck,
Project LISTEN
10
11/7/2015
Carnegie
Mellon
Results
Higher readers speed up less: 3 ms less per grade level.
Longer words speed up more: 2.4 ms more per letter.
Speedup averages 18 ms per encounter (for the first seven).
Speedup averages 30 ms after the first encounter, then 3 ms
less after each subsequent encounter.
Mostow & Beck,
Project LISTEN
11
11/7/2015
Carnegie
Mellon
Context effects
A new practice sentence helps 27 ms more than an old one.
Wide reading beats rereading!
A new practice word pair helps 21 ms more.
A test word pair seen before is 13 ms faster.
Mostow & Beck,
Project LISTEN
12
11/7/2015
Carnegie
Mellon
Are these effects statistically reliable?
Standard errors are 0~3 ms.
But: Linear model ignores dependencies
 Within student (other than proficiency)
 Within word (other than length)
So: Compute p by counting N students, not encounters.
 Restrict to students with 50+ encounters in each condition
 Compare how many students do better in one condition vs. the other.
 Use sign test to compute significance.
New practice sentences beat old ones for 100 of 135 students (p = .000).
 No marked differences between groups
New practice word pairs beat old ones for 151 of 222 students (p = .000).
Mostow & Beck,
Project LISTEN
13
11/7/2015
Carnegie
Mellon
Does word speedup relate to test score gains?
4
2
R = 0.0754
reading gains (years)
3
2
1
0
-20000
0
20000
40000
60000
80000
100000
120000
-1
-2
Summed speedup (ms)
In contrast, # encounters doesn’t predict gains (R2<.001).
Mostow & Beck,
Project LISTEN
14
11/7/2015
Carnegie
Mellon
What affects fluency growth?
1.
Practice on specific words

2.
Practice on specific word pairs

3.
Especially in new sentences
Helps if tested, but new practice pairs are better
Practice in decoding new words

Mostow & Beck,
Project LISTEN
Not analyzed in this talk
15
11/7/2015
Carnegie
Mellon
Conclusions
Wide guided oral reading beats repeated reading!
 Correlational, not experimental; what uncontrolled sample bias?
 Excludes rereading but not recency effects
Relevant to human-guided oral reading?
 Is rereading more motivating for poor readers?
 Wide reading requires more text and more guidance!
Micro-analysis of tutor data finds subtle (ms) effects
 Large, fine-grained, longitudinal sample
 Speech recognition is imperfect
 But unlikely to be biased wrt our variables
Thank you! Questions?
 See papers & videos at www.cs.cmu.edu/~listen.
Mostow & Beck,
Project LISTEN
16
11/7/2015
Carnegie
Mellon
Thanks to fellow LISTENers & friends
Tutoring:




Interns:
Dr. Joseph Beck, mining tutorial data
Prof. Albert Corbett, cognitive tutors
Becky Kennedy, linguist
Joe Valeri, activities and interventions




Listening:
 Dr. Evandro Gouvea, acoustic training
 John Helman, transcriber
 Dr. Mosur Ravishankar, speech recognizer
Programmers:
 Andrew Cuneo, application
Research partners:




DePaul
U. Toronto
U. British Columbia
Ghana
Advisory board
Field staff:




 Kristin Bagwell
 Julie Sleasman
 Dr. Roy Taylor
Grad students:
 Kai-min Chang, LTI
 Cecily Heiner, MCALL
Mostow & Beck,
 Ayorkor Mills-Tettey, RI
Project LISTEN
Alisa Grishman
Brooke Hensler
James Leszczenski
Rachel Minkoff
Kathryn Ayres, children’s stories
Rollanda O’Connor, reading
Charles Perfetti, reading
…
Schools
www.cs.cmu.edu/~listen
17
11/7/2015