Transcript Interactive Knowledge Acquisition Tools: A Tutoring Perspective Yolanda Gil

Interactive Knowledge Acquisition
Tools:
A Tutoring Perspective
Yolanda Gil
Jihie Kim
USC/Information Sciences Institute
August 9, 2002
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
1
Motivation: Investigate Synergies between
Instructional Systems and Acquisition Tools
SOFTWARE
Instructional
System
Acquisition
Tool
?
USER
?
Good
Tutoring
Principles
Good
Learning
Principles
USC INFORMATION SCIENCES INSTITUTE
teaches
teaches
Gil & Kim
2
Our Previous Work in Knowledge Acquisition:
The EXPECT Project at USC/ISI:

EXPECT architecture for knowledge-based systems
exploits highly declarative representations



[Swartout & Gil, KAW-95], [Gil & Melz, AAAI-96] [Blythe et al, IUI01]
http://www.isi.edu/expect
Research focus: interactive knowledge acquisition (KA)
tools that help end users to develop knowledge bases



Deriving models of knowledge interdependencies to detect
knowledge gaps and errors [Kim & Gil, AAAI-99] [Kim & Gil, IUI2000] [Kim & Gil, AAAI-2000]
KA dialogue scripts to guide users by following up on effects of
complex changes [Gil & Tallis, AAAI-97] [Tallis & Gil, AAAI-99]
[Tallis, IJHCS-2001]
Exploiting background theories to understand how new knowledge
fits [Blythe, IJCAI-2001] [Blythe, AAAI-02]
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
3
EXPECT: A User-Centered Framework for
Developing KBSs
EXPECT
Method
instantiator
Domain
dependent
KBS
KBS
compiler
Ontologies and
Method libraries
Knowledge Base
Domain
ontologies
and factual
knowledge
Interdependency
Model (IM)
Dialogue plans
(KA Scripts)
Knowledge-Based
System
USC INFORMATION SCIENCES INSTITUTE
CYC/Sensus
Upper
Problem
solving
methods
Domain
ontologies
Plans
(PLANET)
Evaluations and
Critiques
Resources
(OZONE)
Evaluation
PSMs
KA tools
EMeD
PSMTool
NL Editor
Instrumentation
Gil & Kim
4
Brief Overview of Representative KA
Tools (I)

CHIMAERA [McGuinness et al 2000]



EXPECT [Blythe et al 2001]




Acquisition of task models in Soar
Situated NL instruction is mapped to PSCM [Newell et al. 1991]
KSSn [Gaines & Shaw 1993]



Acquisition of problem solving knowledge
Exploits dialogue scripts, interdependency models, bg k
INSTRUCTO-SOAR [Huffman & Laird 1995]


Acquisition of concepts, relations, instances
Diagnoses faulty definitions
Acquisition of concepts, rules, data
Based on personal construct psychology [Kelly 1955]
PROTOS [Bareiss et al 1990]


Acquisition and classification of new cases
Learning indexes to categories
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
5
Brief Overview of Representative KA
Tools (II)

SALT [Marcus & McDermott 1989]



SEEK2 [Ginsberg et al. 1985]




Acquisition of process models
User interaction based on concept maps [Novak 1977]
TAQL [Yost 1993]



Acquisition of rules
Uses verification and validation techniques
SHAKEN [Clark et al. 2001]


Acquisition of constraints and fixes for configuration design
Exploits Problem Solving Method/ Task (Role-limiting approach)
Acquisition of SOAR rules
Editor for high level language for PSCM [Newell et al. 1991]
TEIREISIAS [Davis 1979]


Acquires and classifies new cases
Learning indexes to categories
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
6
Open Challenges in KA

Users remain largely responsible for the acquisition
process



Decide where, what, when, how, why to enter knowledge
System checks errors, may have some short-term acquisition
goals
Ideally, KA tools should have student-like skills:




Formulate and pursue learning goals
Keep track of lessons and progress
Assess how much they are learning and how useful k is
If teacher is not so great, still capable of learning
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
7
Instructional Systems and Acquisition
Tools: What Are the Synergies?
SOFTWARE
USER
?
Instructional
System
Supplement
Good
Student’s
Tutoring
Principles limitations
Acquisition
Tool
?
Good
Learning
Principles
Supplement
Teacher’s
limitations
USC INFORMATION SCIENCES INSTITUTE
teaches
teaches
Gil & Kim
8
Tutoring and Learning Principles
Relevant to KA [Kim & Gil, ITS 02] (I)
Teaching/Learning principle
Tutoring literature
Start by introducing lesson topics
and goals
Atlas-Andes, Meno-Tutor, Human
tutorial dialog
Use topics of the lesson as a
guide
BE&E, UMFE
Subsumption to existing cognitive
structure
Human learning, WHY, Atlas-Andes
Immediate Feedback
SOPHIE, Auto-Tutor, Lisp tutor,
Human tutorial dialog, human
learning
Generate educated guesses
Human tutorial dialog, QUADRATIC,
PACT
Keep on track
GUIDON, SHOLAR, TRAIN-Tutor
Indicate lack of understanding
Human tutorial dialog, WHY
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
9
Tutoring and Learning Principles
Relevant to KA [Kim & Gil, ITS 02] (II)
Teaching/Learning principle
Tutoring literature
Detect and fix “buggy” knowledge
SCHOLAR, Meno-Tutor, WHY,
Buggy, CIRCSIM
Learn deep model
PACT, Atlas-Andes
Learn domain language
Atlas-Andes, Meno-Tutor
Keep track of correct answers
Atlas-Andes
Prioritize learning tasks
WHY
Limit the nesting of the lesson to a
handful
Atlas
Summarize what was learned
EXCHECK, TRAIN-Tutor, MenoTutor
Provide overall assessment of
learning knowledge
WEST, Human tutorial dialog
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
10
Five Main Functions of KA Tools
KNOWLEDGE
ACQUISITION
BACKEND
ASSIMILATE
INSTRUCTION
USER
INTERFACE
TRIGGER
GOALS
PROPOSE
STRATEGIES
PRIORITIZE
GOALS &
STRATEGIES
Knowledge
Base
USC INFORMATION SCIENCES INSTITUTE
PRESENTATION
DESIGN
Gil & Kim
11
Guidance Exploited by KA Tools
KNOWLEDGE
ACQUISITION
BACKEND
ASSIMILATE
INSTRUCTION
USER
INTERFACE
TRIGGER
GOALS
PROPOSE
STRATEGIES
PRIORITIZE
GOALS &
STRATEGIES
PRESENTATION
DESIGN
USC INFORMATION SCIENCES INSTITUTE
Guidance from
Knowledge Base
Problem Solving
& Task Knowledge
Domain Knowledge
General Background
Knowledge
Example Cases
Guidance from
Meta Knowledge
Knowledge Repres.
Model
Diagnosis &
Debugging Principles
Tutoring & Learning
Principles
Gil & Kim
12
Tutoring and Learning Principles in KA
Tools: Basic Conceptual Framework
USER
INTERFACE
KNOWLEDGE ACQUISITION
BACKEND
Knowledge
Editor
ASSIMILATE
INSTRUCTION
Operational
Principles
TRIGGER
GOALS
Acqu. goals
Acqu. strats
Dialogue
- Goals & Strats
- State
- Suggestions
- History
Knowledge
Base
USC INFORMATION SCIENCES INSTITUTE
PROPOSE
STRATEGIES
PRIORITIZE
GOALS &
STRATEGIES
PRESENTATION
DESIGN
Guess
Generators
General
Tutoring
&
Learning
Principles
Priority
Schemes
Interaction
Guidelines
Gil & Kim
13
Tutoring and Learning Principles
Implicit in KA tools
Tutoring/Learning principle
Assimilate
Instruction
Introduce topics & goals
Use topics of the lesson as
a guide
Subsumption to existing
cog. structure
Immediate feedback
Trigger
Goals
Propose
Strategies
Prioritize
Goals &
Strats
Design
Presentation
EXPECT, SEEK2
SALT
SEEK2
PROTOS
PROTOS
Generate educated guesses
EXPECT
SALT
PROTOS,
SALT
EXPECT
TEIREISIAS
INSTRUCTO-SOAR TEIREISIAS
TEIREISIAS
EXPECT
Keep on track
Indicate lack of
understanding
INSTRUCTO
-SOAR
Detect and fix “buggy” K
TAQL
INSTRUCTOSOAR
EXPECT,CHIMERA
Learn deep models
Learn domain language
Keep track of answers
SEEK2
Prioritize learning tasks
Limit the nesting of lessons
Summarize what is learned
Assess learned knowledge
KSSn
USC INFORMATION SCIENCES INSTITUTE
EXPECT
Gil & Kim
14
Tutoring and Learning Principles in KA
Tools

Observation: Some learning and tutoring principles
are used in some aspects of the dialogue by some
tools
Opportunity: Incorporate principles more thoroughly in all
aspects of the dialogue

Observation: These principles are implicit in the tool’s
code and thus are limited
Opportunity: Exploit declarative representations of learning
state, goals, and strategies
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
15
SLICK (Skills for Learning and
Interactively Capture Knowledge)
USER INTERFACE
KNOWLEDGE ACQUISITION
BACKEND
Proactive
Dialogue
Window
SLICK Dialogue Manager
Awareness
Annotations
KB
Dial.
State History
USC INFORMATION SCIENCES INSTITUTE
KB
Active
Acquisition
Goals
&
Strategies
Gil & Kim
Tutoring
&
Learning
Principles
16
Conclusions

Analysis of existing KA tools shows they use
tutoring/learning principles



Sparsely
Implicitly
Current capabilities of KA tools can be improved by:





Representing tutoring/learning principles declaratively
Organizing the dialogue around lesson topics
Keeping track of how knowledge improves through dialogue
Exposing what knowledge has been assimilated and what
areas need improvement or testing
Assessing their competence and confidence on question
answering
USC INFORMATION SCIENCES INSTITUTE
Gil & Kim
17