Transcript (slides)

In-vivo research on learning
Charles Perfetti
PSLC Summer School 2009
1
In-vivo experiments

In Vitro

In Vivo
PSLC summer school 2009
2
Features of in-vivo
experiments in learning



“On-Line” course?
An Intelligent Tutoring System?
A real class; real students; an
intervention that counts.
PSLC summer school 2009
3
The value of in-vivo
experiments in learning



Noisy, uncontrolled environment
Content of intervention is validated by
course goals
So: Built in generalization to classroom
learning
PSLC summer school 2009
4
Problems faced by an in-vivo
researcher


Noisy, uncontrolled environment
As for your experiment:


Students have other things to do
Instructors have other things to do
PSLC summer school 2009
5
Examples of in-vivo studies


Algebra, Physics, Chemistry, Geometry,
French, Chinese,English
Some with computer tutors in major role



ITS
Practice tutors
Some without tutors or tutors in minor
role
PSLC summer school 2009
6
Pre-requisites for an in-vivo
experiment


Knowledge components analysis
Mapping of KCA to a learning or
instructional hypothesis



Theory based
Empirical precedent
Mapping instructional hypothesis to
specific intervention
PSLC summer school 2009
7
Knowledge
Components vs.
curriculum topic
Single Topic
(Area) as
unit
12 separate
KCs as units
Enabled by Data Shop
Mapping a KCA onto an
instructional hypothesis

The case of Chinese characters
Whole Character
= early morning
Radical = sun
zao3
PSLC summer school 2009
9
Mapping an instructional hypothesis
to an instructional intervention

Learning event space
PSLC summer school 2009
10
Instructional Event Space
Instructional
Events
Explicit or implicit
Focus on Valid Features
Make Knowledge Accessible
Promote Active Processing
Schedule events effectively
Coordinate multiple events
Learning
Events
Assessment
Events
Performance
11
1. Learning meanings of Chinese characters

Whole Character
= early morning
Knowledge Components
Analysis
2 (+2) Knowledge Components:
1.
Radical = sun
2.
zao3

the character as a whole; (plus its
meaning)
the radical that is part of the character
(plus its meaning)
Two approaches based on this
analysis

(1) Dunlap, Liu, & Perfetti; (2) Pavlik

Two different Instructional Events
manipulations

Illustrate 1 here: Feature focus
12
Instructional Event Space
Early
morning
Whole
Character
means x
Default (typical)
Instructional event
Associate
character
form with
meaning
Assessment
Events
Performance
13
Instructional Event Space
Early
morning
Part of character
means x’
Associate radical
with x’ and
whole character
with x
Assessment
Events
Highlighted radical = sun/day
Dunlap et al:
Instructional event
manipulation: semantic
radical instruction
Performance
14
Learning English Spelling
(Background knowledge and feature focusing themes)
Transcriptions of Recorded Speaking Activities
(Excluding Form Errors and Garble)
100%
Proportion of Words in Each Codi ng C ategor y
Correct
80%
Vowel Error
Consonant Error
60%
Multiple C/V Errors
40%
20%
0%
3
4
Arabic
5
3
4
Chinese
5
3
4
Korean
5
3
4
Spanish
5
3
4
Taiwanese
Dunlap, Juffs, Friedline, Perfetti
15
KC analysis of English
spelling





phonology—orthography
/breit/--brate
/hiyl/--heel
/hiyl/--heal
So: phonology-semantics-orthography
16
Feature focusing interventions


130 students in levels 3 4, & 5
Interventions:



“Pure” feature focus: form only
(pronunciation-spelling pairs)
Meaning mediated focus: form + meaning
(pronunciation-meaning-spelling triads)
7 sessions, 30 minutes per session over
7 weeks
17
Audio Dictation Task
1.00
0.90
0.80
0.70
0.60
0.50
Target Items
0.40
Control Items
0.30
0.20
0.10
Dunlap, Juffs,
Friedline,
Perfetti
Spell Check Task
0.00
1.00
0.90
0.80
Pre-Test
Post-Test
All 7 Sessions n = 12
0.70
0.60
0.50
Target Items
0.40
Control Items
0.30
0.20
0.10
0.00
Pre-Test
Post-Test
All 7 Sessions n = 12
Control conditions for in-vivo
experiments

Typical control conditions

Existing classroom instruction


For cog tutors:



Textbook & exercise problems
Another tutoring system
Human tutoring
A control intervention;

2 plausible interventions—which is more effective
PSLC summer school 2009
24
Learning Assessments
1.
2.
3.
Immediate Learning
Long-term retention
Transfer

4.
Over content, form, testing situations
Accelerated Future Learning

New content; learning measure
PSLC summer school 2009
25
Instructional Event Space
Instructional
Events
Explicit or implicit
Focus on Valid Features
Make Knowledge Accessible
Promote Active Processing
Schedule events effectively
Coordinate multiple events
June 2009 NSF Site Visit
Learning
Events
Performance
Assessment
Events
Transfer illustrated:
Liu, Wang, Perfetti Chinese tone
perception study

In-vivo study





Traditional classroom (not online)
Materials from students’ textbook
New materials each week for 8 weeks of term 1
Term 2 continued this, and added novel syllables unfamiliar to the
student
3 instructional conditions




tone number + pin yin, contour + pin-yin; contour only
Hint system
(CTAT) Tutors presented materials in 3 different instructional
interfaces, according to the 3 conditions
Data shop logged individual student data
PSLC summer school 2009
27
Illustration of 2 conditions from
Liu et al
shi
PSLC summer school 2009
28
Data from Liu et al tone study
Scatterplot of Pinyin+Conto, Pinyin+Numbe, Contour Only vs Lesson No.
0
Pinyin+Contour
5
10
Pinyin+Number
15
20
0.8
0.6
Novel
0
1
0.4
0.2
0.0
Contour Only
0.8
Learning curves
week-by-week
0.6
0.4
0.2
0.0
0
5
10
15
20
Lesson No.
PSLC summer school 2009
29
Multiple kinds of transfer

Liu et al shows 2 kinds of materials
transfer


Within term 1, learning sessions, each
syllable to be learned was different but
familiar. So transfer of learning to familiar
items
At second term, there were unfamiliar
syllables. So transfer of learning to
unfamiliar items. (Not so good.)
PSLC summer school 2009
30
Example of acceleration of future learning
(Min Chi & VanLehn)

First probability, then physics. During probability only,

Accelerated
future
learning
Score
Score

Half students taught an explicit strategy
Half not taught a strategy (normal instruction)
Ordinary
transfer
Pre
Post
Probability Training
PSLC summer school 2009
Pre
Post
Physics Training
31
Creating assessments

General strategy:


Include some items from the pre-test


Check for basic learning
Some items similar to training items


Guided by cognitive task analysis (pre-test as
well) including learning goals and specific
knowledge components
Measures near-transfer
Some problems dissimilar to training
problems

Measures far-transfer
PSLC summer school 2009
32
Mistakes to avoid in test
design

Tests that are




Notice problems
in test means
Tests that

Notice
variances
Too difficult
Too easy
Too long
Fail to represent instructed content


Missing content; over sampling from some content
Depend too much on background knolwedge
PSLC summer school 2009
33
Interpreting test results as
learning

Post-test in relation to pre-test. 2 strategies:

ANOVA on

gain scores




First check pre-test equivalence
Not recommended if pre-tests not equivalent
Pre-test, post test as within-subjects variable (t-tests for
non-independent samples)
ANCOVA. Post-tests scores are dependent
variable; pre-test scores are co-variate
PSLC summer school 2009
34
Plot learning results

Bar graphs for instructional conditions


Differences due to conditions
Learning Curves

Growth over time/instruction
PSLC summer school 2009
35
Bar graphs (with error bars!)
PSLC summer school 2009
36
Learning
Scatterplot Curves
of Pinyin+Conto, Pinyin+Numbe,
0
Pinyin+Contour
5
10
Pinyin+N
Error
rate
0.8
Contour Only
Weekly sessions over 2 terms
0.6
PSLC summer school 2009
37
Learning
Scatterplot Curves
of Pinyin+Conto, Pinyin+Numbe,
0
Pinyin+Contour
5
10
Pinyin+N
Error
rate
0.8
Contour Only
Weekly sessions over 2 terms
0.6
PSLC summer school 2009
38
A final word on experiments

In-vivo limitations

The role of (in-vitro) laboratory
studies
QuickTime™ and a
decompressor
are needed to see this picture.
The end
41