Transcript LIPSTICK

LIPSTICK
Creation and Analysis
of a Problem Space
Based on No-Feedback
Exercises
Michael Bellem
Overview






Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
LIPSTICK Demo
Discussion
Overview






Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
LIPSTICK Demo
Discussion
Motivation / Goals






Analyze students’ problem solving behavior
Level of detail: exercise micro structure
Visualize it!
High-level overview + details
Use ActiveMath’s OpenMath-, metadata
Simplify complex problem spaces using
contexts
Overview






Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
LIPSTICK Demo
Discussion
Problem
1.
2.
3.
4.
5.
Extract information from user logs
Build individual problem spaces
Merge problem spaces
Display problem space
Collapse problem space
Problem
Overview



Motivation / Goals
Problem
Solution / LIPSTICK
–
–
–
–
–



Behavior Graphs
Workflow
Node / Edge Merging
Graph Collapsing
Contexts
Evaluation
LIPSTICK Demo
Discussion
Solution / LIPSTICK





Log based
Interactive
Problem Space
Tool Incorporating a
Context Kit
Solution / LIPSTICK

Behavior Graphs
Start
2
1
1+3
1
3+1
1
1
5
4
1
2
Finish
Solution / LIPSTICK

Node attributes:
–
–
–
–
–
–
–
ID
plainFormula
OMFormula
userID
exerciseID
mergeTarget
representedNodes
Solution / LIPSTICK

Edge attributes:
–
–
–
–
–
–
–
ID
userID
exerciseID
traversalCount
accumulatedTraversalCount
mergeTarget
representedEdges
Solution / LIPSTICK

Workflow:
1.
2.
3.
4.
5.
6.
7.
8.
9.
Chose log files to import
Exercise IDs extracted, displayed
Select Exercise ID
Logs filtered and converted to GraphML (via XSL)
GraphML files loaded using prefuse
Subgraphs merged
Interactive graph + statistics displayed
Select context
Collapse graph by merging equivalent nodes
Solution / LIPSTICK
GraphML
+ XSL
+prefuse
XML logs
prefuse
graphs
Merging
prefuse,
custom
merged
prefuse
graph
Collapsing,
contexts
prefuse,
custom
Solution / LIPSTICK

Node / Edge Merging:
–
–
–
Node formulas equivalent  merge
Requires redirecting / merging edges
Want to keep “merge victims”
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
3+1
1
5
1
4
1
4
1
1
Finish
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
3+1
1
5
1
4
1
4
1
1
Finish
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
3+1
1
5
1
4
1
4
1
1
Finish
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
3+1
1
5
1
4
1
4
1
1
Finish
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
1
5
3+1
1
1
4
1
4
1
1
Finish
Solution / LIPSTICK
Start
1
1+3
1
1
1+3
1
1
5
3+1
1
1
4
1
4
2
1
Finish
Solution / LIPSTICK

Graph Collapsing:
–
–
Merge all “equivalent” nodes
Equivalency relation determined by user, e.g.
CAS
Solution / LIPSTICK
Start
3
1+3
2
1
2
5
1
Finish
Solution / LIPSTICK

Contexts:
–
–
–
–
Rule: checks equivalence
Context: set of rules, applied recursively
Programmers can create new rules
Users can define new contexts
Basic Context
Associativity
Commutativity
Distributivity
Overview




Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
–
–
–
–


Basis: No-Feedback Exercises
Graph Collapsing: Quantitative / Qualitative
Runtimes
Improvements
LIPSTICK Demo
Discussion
Evaluation

No-Feedback Exercises:
–
–
–
–
–
–
Winter term 2005
10 students of basic seminar
7 Exercises
No assistance from ActiveMath
Input several steps
Click finish, when you want
Evaluation
Evaluation
Evaluation

Quantitative: Syntactical equivalence
uncollapsed
collapsed
Nodes
Edges
Nodes
Edges
VI
Removed
2
69
68
27
43
67
66
3
51
50
22
35
44
35
4
40
39
24
35
20
63
5
36
35
26
35
10
56
6
30
29
18
21
20
34
7
34
32
20
28
18
50
8
25
24
20
23
6
35
Exercise
Duration
(s)
Evaluation

Associativity
uncollapsed
collapsed
Nodes
Edges
Nodes
Edges
VI
Removed
2
69
68
27
43
67
99
3
51
50
22
35
44
80
4
40
39
24
35
20
66
5
36
35
26
35
10
64
6
30
29
18
21
20
53
7
34
32
20
28
18
57
8
25
24
20
23
6
43
Exercise
Duration
(s)
Evaluation

Commutativity
uncollapsed
collapsed
Nodes
Edges
Nodes
Edges
VI
Removed
2
69
68
26
41
70
99
3
51
50
22
35
44
78
4
40
39
24
35
20
68
5
36
35
26
35
10
58
6
30
29
18
21
20
42
7
34
32
20
28
18
142
8
25
24
20
23
6
109
Exercise
Duration
(s)
Evaluation

YACAS
uncollapsed
collapsed
Nodes
Edges
Nodes
Edges
VI
Removed
2
69
68
26
43
68
99
3
51
50
22
35
44
78
4
40
39
24
35
20
68
5
36
35
26
35
10
58
6
30
29
17
20
22
42
7
34
32
20
28
18
142
8
25
24
20
23
6
109
Exercise
Duration
(s)
Evaluation

Qualitative:
–
Contexts should enable teachers / tutors to find
good representations / simplifications
Evaluation
Evaluation
Evaluation

Runtimes:
–
–
Graph loading / saving fast
Importing logs slower:
Exercise
# of files
Total file
size
Nodes
Edges
Duration
(s)
2
10
11.8 MB
77
80
10.1
3
1
180 KB
36
35
0.2
Evaluation

Improvements:
–
–
–
–
–
–
–
–
–
Performance optimization
More stable CAS (or better usage?)
More equivalency rules
Different system of contexts?
Better formula renderer (or better usage?)
Exercise generation
Manual editing of the behavior graph
Port to Linux
Real-world evaluation
Overview






Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
LIPSTICK Demo
Discussion
Overview






Motivation / Goals
Problem
Solution / LIPSTICK
Evaluation
LIPSTICK Demo
Discussion
Discussion


Questions?
Remarks?
 Thank You
Additional Topics

Related Work – Behavior Recorder:
+
+
+
+
+
-
Carnegie Mellon University
Displays behavior graphs
Live behavior recording from Cognitive Tutors
Exercise generation
Nice UI and graph layout
No complex formulas from Open Math
Hard to import ActiveMath logs