Transcript A Trace-based Framework for Analyzing and Synthesizing Educational Progressions Erik Andersen Sumit Gulwani Zoran Popović.

A Trace-based Framework for Analyzing and
Synthesizing Educational Progressions
Erik Andersen
Sumit Gulwani
Zoran Popović
Procedural Learning
21
+22
43
Software Learnability
Dong et al. CHI 2012
Challenge
Flow
Anxiety
Boredom
Skill
M. Csikszentmihalyi
Zone of Proximal Development
can’t do yet
can do with guidance
can do now
L. Vygotsky
Procedural Traces
• What are traces?
• What can we do with traces?
same # of digits?
NO
greater # of digits?
NO
report “less”
YES
YES
first digit same?
report “greater”
NO
first digit greater?
NO
report “less”
YES
YES
second digit same?
report “greater”
NO
second digit greater?
YES
YES
...
report “greater”
NO
report “less”
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
first digit same?
report “greater”
NO
first digit greater?
H
NO
YES
D
YES
second digit same?
report “greater”
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
L
G
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
L, H
G
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
L, H, DS
G
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
L, H, DS, DG
G
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
G
L, H, DS, DG, DDS
report “greater”
NO
same # of digits?
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
L, H, DS, DG, DDS, DDG…
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
Compare 313 to 324.
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
Compare 313 to 324.
What can we do with traces?
• Rank their difficulty
• Analyze and compare progressions
• Synthesize new progressions
What can we do with traces?
• Rank their difficulty
• Analyze and compare progressions
• Synthesize new progressions
Partial Ordering
• Loop executes longer = harder
– more digits in number comparison
– more columns in subtraction
• Exceptional conditions = harder
– if boolean expression: statement
– borrowing in subtraction
– carrying in addition
n-grams
1-grams
the boy threw the ball
2-grams
the boy threw the ball
n
1
2
3
n-grams
H, L, D, S, G
DD, DS, DG
DDS, DDG
n-gram based Partial Ordering
for some integer n
if T1 contains all of the n-grams of T2
then T1 > T2
1-grams → statement coverage
∞-grams → path coverage
What can we do with traces?
• Rank their difficulty
• Analyze and compare progressions
• Synthesize new progressions
if length(n1) < length(n2): return less
if length(n1) > length(n2): return greater
for i ← 0 to length(n1) – 1:
if n1[i]< n2[i]: return less
if n1[i]> n2[i]: return greater
return equal
L
H
D
S
G
JUMP Math
JUMP Math
Singapore Math
JUMP Math
Singapore Math
What can we do with traces?
• Rank their difficulty
• Analyze and compare progressions
• Synthesize new progressions
if length(n1) < length(n2): return less
if length(n1) > length(n2): return greater
for i ← 0 to length(n1) – 1:
if n1[i]< n2[i]: return less
if n1[i]> n2[i]: return greater
return equal
DDS
11 < 12
L
H
D
S
G
Problem
Addition
Fraction Computation
Fraction Reduction
Integer Comparison
Subtraction
Prime Factorization
Time
64 min
16 min
2 min
2 min
2 min
2 min
Unique Traces
1433
72
29
26
157
70
DragonBox
DragonBox
Validating Our Partial Ordering
•
•
•
•
20 participants
20 level pairs
Users ranked pairs: easier, harder, the same
Calculated Root-Mean-Squared-Error
– our ordering: 0.35
– random baseline: 1.00
Summary
new representation of problem difficulty
allows us to:
– rank problem difficulty
– analyze and compare progressions
– generate an entire progression
pragmatic way to make content fit theory
Acknowledgments
• Help: David Yamanoha and Ryoko Nozawa
• Funding: Bill and Melinda Gates Foundation,
Office of Naval Research, Hewlett Foundation,
Adobe, and Intel
• Contact me: [email protected]
From here
• automate other things
– tutorials
– diagnosis of student errors
Challenge
Flow
Anxiety
Boredom
Skill
Implications
• Games
• Education
“Games are a series of interesting decisions.”
– Sid Meier
Artifacts
Artifacts
Progressions
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
Compare 32 to 314.
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
report “greater”
NO
first digit same?
first digit greater?
H
NO
YES
D
YES
report “greater”
second digit same?
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
G
DDDDG 1112 > 1111
Problem
Addition
Book
SS:M
SMS
Fraction
SS:M
Addition & SMS
Subtraction JUMP
Fraction
SS:M
Reduction JUMP
Integer
SS:M
Comparison JUMP
Problems
584
576
66
248
131
66
56
24
88
Traces
17
48
13
15
11
13
11
6
12
% gen
88%
64%
100%
100%
95%
100%
100%
100%
100%
•
•
•
•
HCI
education
software engineering
video games
Promise
new representation of problem difficulty
can
analyze problem sets
generate an entire progression
Promise
new representation of problem difficulty
can
analyze problem sets
generate an entire progression
if length(n1) < length(n2): return less
if length(n1) > length(n2): return greater
for i ← 0 to length(n1) – 1:
if n1[i]< n2[i]: return less
if n1[i]> n2[i]: return greater
return equal
L
H
D
S
G
same # of digits?
NO
greater # of digits?
NO
report “less”
YES
YES
first digit same?
report “greater”
NO
first digit greater?
NO
report “less”
YES
YES
second digit same?
report “greater”
NO
second digit greater?
YES
YES
...
report “greater”
NO
report “less”
same # of digits?
NO
greater # of digits?
NO
report “less”
L
report “less”
S
report “less”
S
YES
D
YES
first digit same?
report “greater”
NO
first digit greater?
H
NO
YES
D
YES
second digit same?
report “greater”
NO
second digit greater?
G
NO
YES
D
YES
...
report “greater”
DDS
G
11 < 12
Solver
action1
action2
action3
...
DragonBox Level
A
AB
AA
AAB
ABC
n-gram based Partial Ordering
• the boy passed the ball
– unigrams: the, boy, passed, ball
– bigrams: the boy, boy passed, passed the, the ball
– trigrams: the boy passed, boy passed the, passed
the ball
n-gram based Partial Ordering
• Integer Comparison
– unigrams: H, L, D, G, S
– bigrams: DD, DG, DS
– trigrams: DDG, DDS
Promise
new representation of problem difficulty
can
analyze problem sets
generate an entire progression
Promise
new representation of problem difficulty
can
analyze problem sets
generate an entire progression
Integer Addition
Fraction Addition and Subtraction