Transcript Building Cognitive Model for Cognitive Tutors by Demonstration

SimStudent:
A computational model of learning for
Intelligent Authoring and beyond
Noboru Matsuda
Human-Computer Interaction Institute
Carnegie Mellon University
CTAT: Cognitive Tutor Authoring Tools
• Example-Tracing Tutor
– Zero programming
– A cognitive model specific to a particular problem
– Limited generalization by editing a behavior graph
SimStudent Project
• Model-Tracing Tutor
– Requires a cognitive model
– Cognitive task analysis is hard
– Writing production rules is even more challenging
• Performing a task is much easier…
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
2
Next Generation Authoring
Build a tutor GUI
Teaching a solution
SimStudent Project
SimSt. learning
Production Rules
Rule simplify-LHS:
Rule simplify-LHS:
IF
is-equation( Eq ),
IFRule simplify-LHS:
is-equation(
Eq ),
is-lhs(
Eq, Lhs ),
is-lhs(
Eq,
IF polynomial(
is-equation(
Lhs Lhs
), ),Eq ),
polynomial(
Lhs
all-var-terms(
is-lhs( Lhs
Eq, )),
Lhs ),
all-var-terms(
Lhs
Then
simplify(
polynomial(
Lhs, S-lhs
Lhs
),) ),
Then
simplify(
), )
enter(
S-lhs
all-var-terms(
)Lhs, S-lhsLhs
enter(
S-lhs ) Lhs, S-lhs ),
simplify(
enter( S-lhs )
Then
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
3
SimStudent
• Machine learning agent
– Learns problem-solving steps by …
– Observes model solutions / solving problems, and …
– Outputs a set of production rules
SimStudent Project
• Fundamental technology
– Programming by Demonstration
– Inductive Logic Programming
Lau & Weld (1998). Blessing (1997).
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
4
Authoring Strategies
• Authoring by demonstration
– Demonstrate whole solutions
– Learning from worked-out examples
• Demonstrated solutions as positive examples
SimStudent Project
• Authoring by tutoring
– Interactively tutor with immediate feedback and hint
– Learning by tutored problem-solving
• Learning by generalizing hint with taking feedback into
account
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
5
Demo!
Authoring by Tutoring
SimStudent Project
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
6
Learning Production Rules in 3 parts:
What-When-How
If
such and such constraints hold
When
SimStudent Project
among this and that GUI elements
What
Then
do actions with the GUI elements
How
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Production Rule in JESS
(defrule trans-lr-lhs
GUI elements
Working Memory Element (WME)
?problem <- (problem (interface-elements ?table1 ? ? ?))
?table1 <- (table (columns ?column1))
?column1 <- (column (cells $?m1 ?cell0 $?))
?cell0 <- (cell (value ?val0&~nil))
WME path
LHS
?problem <- (problem (interface-elements ? ?table2 ? ?))
?table2 <- (table (columns ?column2))
?column2 <- (column (cells $?m2 ?cell1 $?))
?cell1 <- (cell (value ?val1&~nil))
SimStudent Project
?column1 <- (column (cells $?m3 ?cell2 $?))
?cell2 <- (cell (name ?selection) (value ?input))
(test
(test
(test
(test
(test
(test
(consecutive-row ?cell0 ?cell2))
(same-column ?cell0 ?cell2))
(distinctive ?cell0 ?cell2))
(consecutive-row ?cell1 ?cell2))
(same-column ?cell1 ?cell2))
(distinctive ?cell1 ?cell2))
(test (polynomial ?val0))
(test (not (has-var-term ?val1)))
Topological
constraints
Feature
constraints
WME
conditions
Constraints
=>
(bind ?input (first-var-term ?val0))
Actions
(modify ?cell2 (value
?input)) )
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
RHS
Background Knowledge
• Domain concepts to “explain” demonstrations
– Operators
– Feature predicates
• External Jess function written in Java
SimStudent Project
(defrule multi-lhs
…
?var22140 <- (column (cells ? ? ?var22143 ? ? ? ? ?))
?var22143 <- (cell (value ?val0&~nil))
=>
)
(test (fraction-term ?val0 ))
(bind ?val2 (denominator ?val0))
(bind ?input (mul-term-by ?val0 ?val2))
…
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
9
Example： Algebra domain
• 16 Feature predicates & 28 operators
SimStudent Project
Feature Predicates for LHS conditions
Operators for RHS actions
HasCoefficient
HasConstTerm
HasVarTerm
Homogeneous
IsFractionTerm
IsConstant
IsDenominatorOf
IsNumeratorOf
IsPolynomial
Monomial
NotNull
VarTerm
IsSkillAdd
IsSkillSubtract
IsSkillDivide
IsSkillMultiply
AddTerm
AddTermBy
Coefficient
CopyTerm
Denominator
DivTerm
DivTermBy
EvalArithmetic
FirstTerm
FirstVarTerm
GetOperand
InverseTerm
LastConstTerm
LastTerm
LastVarTerm
MulTerm
MulTermBy
Numerator
ReverseSign
RipCoefficient
SkillAdd
SkillClt
SkillDivide
SkillMultiply
SkillRf
SkillMt
SkillSubtract
VarName
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
10
Learning Results
SimStudent Project
% Correct rule firings (10 test tasks)

0, if C  0
C
StepScore  

 C  I
Authoring by
tutoring

Better
than
Authoring by
demonstration
# of training tasks
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
11
Authoring Time
Authoring by Tutoring
• Authoring by tutoring
took 86 minutes
• Authoring by demonstration
took 238 minutes
SimStudent Project
•
A 2.8x speed-up!
Authoring by Demonstration
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
12
Example: Stoichiometry Tutor
SimStudent Project
Approximate Time Distribution (hours)
30
40
Operator generation
Feature predicate generation
Debugging
Testing
55
15
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
13
SimStudent Projects
• Intelligent Authoring
– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation
SimStudent Project
– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner
– Learning by teaching
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
14
SimStudent Projects
• Intelligent Authoring
– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation
SimStudent Project
– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner
– Learning by teaching
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
15
Model of Incorrect Learning
SimStudent Project
• Identify errors students
commonly make
• Weaken SimStudent’s
background knowledge
• Let SimStudent make an
induction error
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
16
Weak Prior Knowledge Hypothesis
• Multiple ways to make sense of examples
strong prior knowledge
Get a coefficient and divide
SimStudent Project
3x=5 “divide by 3”
Get a denominator and multiply
4/x=5
“multiply by x”
“divide by 4”
Get a number and divide
weak prior knowledge
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
17
Target Common Errors
ax  b  c  "add b"
ax  b  c  "subtract b"
SimStudent Project

ax  b  "multiply by a"


x
 b  "divide by a"
a
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
18
Results: Learning Rate
Step Score
SimStudent Project
# training problems
Steps Score =
0 (if there is no rule applicable)
# correct rule applications / # all rule applications
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
SimStudent Projects
• Intelligent Authoring
– Building a Cognitive Tutor as a CTAT Plug-in
• Student Modeling and Simulation
SimStudent Project
– Controlled educational studies
– Error formation study
– Prerequisite conceptual knowledge study
• Teachable Peer Learner
– Learning by teaching
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
20
Learning by
Teaching
SimStudent
SimStudent Project
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
Demo!
SimStudent Project
Learning by Teaching
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
22
Learn more about SimStudents
• Project Web
– www.SimStudent.org
• Download & Tutorial
– http://ctat.pact.cs.cmu.edu (linked from project web)
SimStudent Project
• Contact us!
– [email protected]
PSLC Summer School 2010 :: SimStudent Demo :: Noboru Matsuda (CMU)
23