Transcript Cognitive Tutors

Robust learning of vocabulary in
classrooms and in CALL
Alan Juffs, Lois Wilson
University of Pittsburgh
Maxine Eskenazi Michael Heilman
CMU
AAAL, ,Washington DC, March 31, 2008
PSLC ELI CMU Pitt
1
Funding
This research is supported by the
United States of America National Science Foundation
Grant: SBE-0354420
English Language Institute (ELI)
University of Pittsburgh
Language Technologies Institute (LTI)
Carnegie Mellon University
7/16/2015
PSLC ELI
2
Pittsburgh Science of Learning Center


Exploring learning in math, science and
second language learning
Attempting a general theory of robust
learning; robust learning should:
• Be long-term
• Transfer to new contexts
• Accelerate future learning


VanLehn (2006, p. 5)
The English Language Institute at the University
of Pittsburgh is one of seven site partners with
the PSLC that provide in vivo research contexts
7/16/2015
PSLC ELI
3
The problem:
Vocabulary and reading in an Intensive English Program
(IEP)


Goal of IEP reading: prepare for
academic reading
Assume Nation: 1000/2000 lists



http://www.lextutor.ca
http://www.vuw.ac.nz/lals/staff/paulnation/nation.aspx (Nation, 2007, webcast)
Target vocabulary: Coxhead word list

7/16/2015
http://www.vuw.ac.nz/lals/research/awl/info.html
PSLC ELI
4
Reading Curriculum and vocabulary
(Nation, 2001; 2007)





Issue: how best to develop fluency and
refinement in these AWL items and
enhance long-term retention
Students’ books inadequate
Students don’t do extensive reading out of
class
Class has heterogeneous L1
Students need to get from 2000 level
frequency to 4-5000 frequency for
academic readiness in one year
7/16/2015
PSLC ELI
5
REAP - LTI at CMU


REAP (REAder Specific Practice; Brown and Eskenazi, 2004; CollinsThompson and Callan, 2004) <http://cs.reap.cmu.edu>.
i) a search engine that finds text passages satisfying very specific
lexical constraints:
• Filters: the system is set to filter for AWL words in texts of L1 English
grades 6-8 level reading.
• Data base of 50,000 documents (after filtering from 10 million)
• Each document about 1000 words
• Total words: 50,000,000 C.f Cambridge and Nottingham Spoken
corpus of 5,000,000 (McCarthy, 2006)


ii) selecting materials from an open-corpus (the Web), thus
satisfying a wide range of student interests and classroom needs,
and
iii) the potential to model an individual’s degree of acquisition and
fluency for each word in a constantly-expanding lexicon so as to
provide student-specific practice and remediation
7/16/2015
PSLC ELI
6
Solution to the problem?


REAP = researchers/teachers think of this as a
way for individuals acquiring the AWL word
list in a rich context through extensive reading
that will also reinforce known words.
But issues of focus on the task
• Juffs et al. (submitted) found that students
transformed the CALL tasks, either skipping
reading or focusing too much on individual words
7/16/2015
PSLC ELI
7
What we know about vocabulary
learning from recent ‘lab’ studies

Hulstijn and Laufer (2001)
• Advanced ESL in Holland and Israel
• A) M-choice- B) fill in the blank C) writing
10 words floor effects in delayed post-test, but superior
effects for writing practice and fill in the blank

Folse (2006)
• IEP context in the USA
• Gains small for all conditions
• Fill in the blank ‘most efficient’ compared to writing

Barcroft (2004, 2006)
• Beginning Spanish L2
• Writing distracts from learning forms in first exposure
7/16/2015
PSLC ELI
8
Research Questions





What does vocabulary learning look in ‘in vivo’ as
opposed to a very tightly controlled study?
(action research)
How does in class vocabulary instruction differ
from the CALL in the reading course in
intermediate ESL?
How do the learning outcomes differ?
If there are differences, what might the source of
those differences be?
Can deeper processing through writing be
‘skipped’ by using a CALL program?
7/16/2015
PSLC ELI
9
Procedure with Software








Take a pre-test to establish individual focus word list of 100
Coxhead words.
Read texts selected to provide examples of the words they
don’t know. While reading, may choose to get help by
clicking (or not) on words and looking at definitions.
‘Reading check’ question after reading the passage
Vocabulary check question.
If target word wrong, another passage with that word in
presented right away; otherwise word tested later
All words seen at least 5 times, in the text, in the
vocabulary questions and reading check.
40 minutes per week in class with the REAP system.
End of term post-test cloze and production tests.
7/16/2015
PSLC ELI
10
Example text: highlights
7/16/2015
PSLC ELI
11
Procedure in Class








Words chosen by curriculum supervisor on the basis of her
focus/assessment
Take a pre-test to establish individual focus word list of 58
AWL words.
All students given the same list to study. Quizzes during the
term.
Look-ups in class, homework exercises, student board work
In class instruction based on pair/ group work and
interaction and written practice
40-50 minutes a week of in class and homework.
Final post-test in cloze format.
Final post-test of production
7/16/2015
PSLC ELI
12
Participants and MTELP
Language N Fall
Mean
African
2 58.5
Arabic
18 46.89
Chinese 4 63.75
Japanese 1 56.00
Korean
6 60.83
Spanish 2 68.00
Thai
1 53.70
Turkish
0 .
TOTAL 34 53.70
7/16/2015
Fall
SD
19.09
7.78
12.12
.
13.98
0
.
.
12.08
PSLC ELI
N
Spring
Mean
0 .
10 49.30
7 59.29
2 53.00
13 60.34
2 75.00
.
.
4 61.75
38 57.84
Spring
SD
.
11.81
11.85
8.49
13.13
8.48
.
7.04
12.73
13
Results




Pre-test
Post-tests
Average gain
Analysis of long-term transfer in
writing in the database.
• By BNC
• By AWL
• http://www.lextutor.ca (T. Cobb)
7/16/2015
PSLC ELI
14
7/16/2015
PSLC ELI
Pr
od
uc
t
io
n
%
ga
in
se
en
10
RE
AP
ra
ge
Av
e
ds
W
or
1 3 .1 7
RE
AP
RE
AP
pr
et
es
t
%
0
Lo
ok
up
s
RE
AP
30
tio
n/
du
c
ga
in
20
Ta
rg
et
Pr
o
ag
e
Av
er
0
tt
es
t/
4
Po
s
58
te
st
/
Pr
e-
Fall 2 0 0 7
80
7 3 .2 8
70
60
50
40
Fall 2 0 0 7
3 2 .1 7
30
29
19
2 2 .2 7
1 4 .4 1
9 .2 4
0
15
Fall 07: Words used in writing

Focus words seen in REAP: 10 uses

BNC-1,000 [ fams 2 : types 2 : tokens 2 ] assume produce

BNC-2,000 [ fams 1 : types 1 : tokens 1 ] distinction

BNC-3,000 [ fams 1 : types 1 : tokens 1 ] conceive

BNC-4,000 [ fams 1 : types 1 : tokens 1 ] abandon

BNC-5,000 [ fams 1 : types 1 : tokens 1 ] derive

BNC-6,000 [ fams 2 : types 2 : tokens 2 ] cite (x3) prohibit
7/16/2015
PSLC ELI
16
Fall 07 Words used from class: 840










BNC-1,000 [ fams 14 : types 15 : tokens 15 ] approach basis client effect
effective functional key notice point present process regarding specific tend
BNC-2,000 [ fams 18 : types 19 : tokens 19 ] accessible alternative analysis
analyze challenge consistent engage essential focus gather image interpret
monitor option promote recruit reserve status ultimately
BNC-3,000 [ fams 6 : types 6 : tokens 6 ] peak prominent randomly speculate
suspend unique
BNC-4,000 [ fams 5 : types 5 : tokens 5 ] conduct objective profile rigid trend
BNC-5,000 [ fams 4 : types 4 : tokens 4 ] cultivate emerge resemble ritual
BNC-6,000 [ fams 1 : types 1 : tokens 1 ] thrive
BNC-7,000 [ fams 1 : types 1 : tokens 1 ] parameter
BNC-9,000 [ fams 1 : types 1 : tokens 1 ] overstate
BNC-11,000 [ fams 1 : types 1 : tokens 1 ] adept (x11)
BNC-14,000 [ fams 1 : types 1 : tokens 1 ] admonish (x15)
7/16/2015
PSLC ELI
17
Fall 07 AWL Words









Sublist 1
analysis 2 analyze 3 approach 5 consistent 3 functional 17 interpret
4 process 48 specific 37
Sublist 2 conduct 3 focus 49
Sublist 3 alternative 21
Sublist 4 accessible 1 emerge 2 option 21 parameter 11 promote 5
status 37
Sublist 5 challenge 21 image 11 monitor 27 objective 11 trend 8
Sublist 7 ultimately 1 unique 12
Sublist 8 randomly 2
Sublist 9 rigid 20 suspend 15
7/16/2015
PSLC ELI
18
Spring 2 0 0 7
90
7 8 .1
80
70
60
5 7 .1 8
50
Spring 2 0 0 7
40
3 5 .5 7
30
20
1 7 .6 4
17
14
9 .6 8
10
PSLC ELI
Po
st
-t
es
t
D
el
ay
e
d
ga
in
Av
er
ag
e
RE
AP
pr
et
es
t
RE
AP
Ta
rg
et
Pr
o
du
ct
io
n/
30
%
ga
in
Av
er
ag
e
0
tt
es
t/
4
Po
s
58
-t
es
t/
Pr
e
7/16/2015
Lo
ok
up
s
0
0
19
Relationship between proficiency
and learning

Matthew Effect:
• Stanovich (1986)
• Parable of the Talents
• Matthew 25:29

7/16/2015
For everyone who has will be given more,
and he will have in abundance. Whoever
does not have, even what he has will be
taken from him.
PSLC ELI
20
MTELP Fall class

Predicts in class
• Pretest:
• Post-test:
• Production:
r= 0.52, p ≤.01
r= 0.59, p ≤ .01
r= 0.58, p ≤.01

Predicts REAP

• Cloze:
r= 0.53, p ≤.01
• Production:
r= 0.59, p ≤.01
Not related to look ups or amount read
7/16/2015
PSLC ELI
21
MTELP Spring class

Predicts in class
• Pretest:
• Post-test:
• Production:

Predicts REAP
• Cloze:

r= 0.63, p ≤.05
r= 0.36, p ≤ .06, ns
r= 0.44, p ≤.02
r= 0.58, p ≤.001
Not related to look ups or amount read
7/16/2015
PSLC ELI
22
Partial Correlations

Controls for MTELP score
• Fall class


Class and Reap Production, r=0.61, p ≤.01
Class and REAP cloze, r= 0.42, p ≤ .05
• Independent of overall proficiency, the CALL
and classroom learning is similar: students
who work hard in class, and work with the
computer,achieve similar gains.
• The pre-test on in class vocabulary does NOT
correlate with gains in REAP or in class when
MTELP is controlled for.
7/16/2015
PSLC ELI
23
Discussion

Hulstijn and Laufer (2001)
• Involvement load hypothesis

Need: zero--external

Teacher -- CALL
Search: zero- provided 

0- -
internal [cognitive?]
lookup
CALL - teacher/class -- Self/CALL
Evaluation
• Non-linguistic - CALL multiple choice
• Fill in blank- in class practice
• Free production - in class homework and practice
7/16/2015
PSLC ELI
24
Conclusions

Students who have more going in, learn
more on the whole.
• Key importance of 1000 and 2000 level words


Written output practice, combined with
involvement in class, may produce more
learning than CALL programs that provide
exposure only even if students are
‘internally motivated’ by self-selecting lists
Writing may take more time, but that may
be what it takes for robust learning of new
vocabulary
7/16/2015
PSLC ELI
25
Conclusions

Folse (2006)
• Writing is time consuming compared to fill-inthe blank
• Writing may be time consuming, but for robust
learning written production is key.

Barcroft (2006)
• Initial exposure, writing may compete for
resources
• For subsequent learning, writing is key
7/16/2015
PSLC ELI
26
Conclusions

Support general point by Allum
(2002)
• Students like computers but prefer
classroom activities when the same
materials are used.
• Juffs et al: students need training on
how to use software.
7/16/2015
PSLC ELI
27
Conclusions

Supports recommendations for online interactivity



Zapata and Sagarra (2007)
Recommend on line vocabulary work book
and feedback
Need to select important vocabulary
carefully and focus on it
• Externally provided vocabulary lists
have face validity
7/16/2015
PSLC ELI
28
PSLC ELI
29
References







Allum, P. (2002). CALL and the classroom: the case for comparative research.
ReCALL, 14, 146-166.
Barcroft, J. (2004). Effects of sentence writing in second language lexical acquisition.
Second Language Research, 20, 303-334.
Barcroft, J. (2006). Negative Effects of forced output on vocabulary learning. Second
Language Research, 22, 487-497.
Folse, K. S. (2006). The effect of type of written exercise on L2 vocabulary retention.
TESOL Quarterly, 40, 273-293.
Hulstijn, J., & Laufer, B. (2001). Some empirical evidence for the involvement load
hypothesis in vocabulary acquisition. Language Learning, 51, 539-558.
Juffs, A., Friedline, B. F., Eskenazi, M., Wilson, L., & Heilman, M. (in review). Activity
theory and computer-assisted learning of English vocabulary. Applied Linguistics.
Stanowicz, K. E. (1986). Matthew effects in reading: some consequences of
individual differences in the acquisition of literacy. Reading Research Quarterly, 21,
360-407.
7/16/2015
PSLC ELI
30