Transcript presentation

TEMPLE:
TEMPLate Extension Through
Knowledge Acquisition
Yolanda Gil
Jim Blythe
Information Sciences Institute
University of Southern California
http://www.isi.edu/expect
{gil, blythe}@isi.edu
USC INFORMATION SCIENCES INSTITUTE
EXPECT
1
Acquiring Planning Knowledge

Problem: SOF users need to add knowledge to these
planning tools





Approach: provide knowledge acquisition tools to adapt
and extend pre-existing planning knowledge




ROEs, commander’s guidance
Plan evaluation/critiquing criteria
Highlight the information that is important to them
Add/extend templates
Exploit ontologies and background knowledge so users don’t
have to start from scratch
KA Scripts guide the user through multiple steps
Users manipulate English paraphrases of internal representations
Benefits:

Users can extend the tool’s baseline knowledge for the operation
USC INFORMATION SCIENCES INSTITUTE
EXPECT
2
Prototype for adding plan critiques:
Expect’s PSM Tool
User adds detailed
knowledge through
English paraphrases
Questions formulated
based on background
knowledge
USC INFORMATION SCIENCES INSTITUTE
EXPECT
3
The next 100 days

Allow users to specify and customise “sentinels” that
check for new information and alert planners when
needed.

Our tools generate Java.

Extend ontologies and background knowledge to
handle SOF domain.

Integrate with one of the jumpstart applications,
probably the travel planning tool, using InterAcT.
USC INFORMATION SCIENCES INSTITUTE
EXPECT
4
Backup slides

Description of approach, tools and experiment from
HPKB project.
USC INFORMATION SCIENCES INSTITUTE
EXPECT
5
Key Technologies



Guiding users through knowledge acquisition scripts
[Tallis and Gil 99] that capture typical dialogues that
users follow to enter new knowledge step by step
Exploiting domain-independent background
knowledge about plan evaluation and critiquing
[Blythe & Gil 99] that use background knowledge
about plan evaluation and critiquing to guide the
dialog
An English-based editor [Blythe & Ramachandran 99]
that lets the user add or modify internal knowledge by
manipulating its English paraphrase, without having
to see or understand the internal formal representation
USC INFORMATION SCIENCES INSTITUTE
EXPECT
6
Architecture of TEMPLE
Temple
Constraint
AcquisitionUI
(Client)
Constraint
wizard
(Server)
English
editor
SOF
Backgroundknowledge
Acquisition
Scripts
Natural
Language
Generator
Constraint
viewer
ActiveTemplatesToolkit
USC INFORMATION SCIENCES INSTITUTE
Ontologies
¤ constrainttypes
¤ actionsandplans
¤ proactive
suggestions
Methodbase
Compiler
Template
library
Domain
knowledge
Domain
models
and
templates
Domain
constraints
Domain
methods
Executable
constraints
EXPECT
7
Evaluation and Critiquing Knowledge
Submethods
for checking
plan resources
Plan
ontology
(PLANET)
Submethods
for checking
plan structure
Ontology of
critiques
Ontology of
resources
Domain-specific
critiques
Reused
knowledge
(ontologies
and methods)
Domain-specific
knowledge
Domain-specific
plan critiquing and evaluation system
USC INFORMATION SCIENCES INSTITUTE
Domain-specific
submethods
EXPECT
8
An Ontology of Plan Evaluation Criteria

Captures general knowledge of how to evaluate
plans with respect to standard norms of plan
development
ako
ako
Ill-formed
description
Statement
critique
Complete statement
Correct statement
isa
Does <unit> have sufficient combat
power to accomplish its mission?
Clear statement
ako
Link
critique
...
Correct link
isa
Does the purpose of supporting effort X
support the main effort?
ako
isa
Structure ako
Complete
plan
critique
...
USC INFORMATION SCIENCES INSTITUTE
Does the COA include a statement
of the reserve forces?
EXPECT
9
KA Scripts



Helps the user add new critiques by using a background
theory of plan evaluation and critiquing.
KA Scripts guide the dialog with the user about the new
critique (wizard-type interaction).
The tool creates some of the needed methods for the
critiques, helps the user to create new ones (by
suggesting initial templates), and ensures consistency
with existing knowledge.
USC INFORMATION SCIENCES INSTITUTE
EXPECT
10
English-Based Editor
Generates automatically English paraphrases of
problem-solving fragments, and presents alternative
text to replace parts of
the paraphrase based
on the ontologies and
background knowledge
NL description of
method
Alternatives for
selected text fragment
USC INFORMATION SCIENCES INSTITUTE
EXPECT
11
First experiment: An ablation test on the
PSM-Based KA Scripts




Hypothesis: the PSM-Based KA Scripts significantly
reduce the expertise and the effort required to add a
new critique to the knowledge base.
KA tasks: add two new critiques to the EXPECT COA
critiquer (a completeness check and a resource check)
Knowledge (and tool) ablation experiment: Two
tasks done using PSM-Based KA Scripts, two tasks
done without
Subjects: four Army officers, previously trained on
EXPECT’s language for a day
USC INFORMATION SCIENCES INSTITUTE
EXPECT
12
Sample Tasks Given to Subjects

Simple critique: Add a critique to check if the COA
has a security statement.

Complex critique: Add a critique to check if each task
in the COA has sufficient force ratio.
To compute force ratio, divide the sum of combat
powers of the Blue units assigned to the task by the
sum of combat powers of the Red units acted on by
the task.
(Two other comparable tasks were also used)
USC INFORMATION SCIENCES INSTITUTE
EXPECT
13
Quantitative results: what users could do

Users could complete more tasks using the PSMbased KA scripts
35
30
25
20
with PSMTool
ab late d versi on
15
10
5
0
ea sier
ta sk 1
ea sier
ta sk 2
mo re
compl ex
ta sk 1
mo re
compl ex
ta sk 2
Al l
LEGEND:
indicates total tasks
USC INFORMATION SCIENCES INSTITUTE
EXPECT
14
Quantitative results: speed improvements

Time reduction using the PSM-based KA Scripts
Time in
minutes
20
18
16
14
12
10
8
6
4
2
0
ablated
version
with scripts
Completeness
Resource
critique
check critique
(simpler)
(more complex)
USC INFORMATION SCIENCES INSTITUTE
EXPECT
15
Axiom acquisition rates:
Experiment with PSM-Based KA Scripts
with PSM-Based
KA Scripts
Adding small amounts
of new knowledge
2.12 ax/min
Adding larger amounts
of new knowledge
1.26 ax/min
USC INFORMATION SCIENCES INSTITUTE
with ablated version
1.1 ax/min
N/A (users were not
able to do tasks)
EXPECT
16
Summary




Using the PSM-Based KA Scripts significantly
reduced the time taken to add a critique
Using the PSM-Based KA Scripts, all four subjects
successfully added simple critiques to the EXPECT
critiquer; three of them successfully added more
complex critiques.
Without the PSM-Based KA Scripts, three out of four
subjects successfully added simple critiques and two
added more complex critiques.
Comments on the tool usability were positive in all
cases.
USC INFORMATION SCIENCES INSTITUTE
EXPECT
17
Second experiment with PSM-Based KA
Scripts and English-Based Editor




Hypothesis: the combination of the PSM-Based KA
Scripts and English-based editor allows a user with very
little training to add new critiques.
Single subject usability test: A subject was briefed in COA
critiquer and the KA interface (but not about EXPECT)
for 20min and asked to add two critiques using the tool
KA tasks: add two new critiques to the EXPECT COA
critiquer (a completeness check and a resource check),
same used in the previous experiment
Subject: an Army officer with no EXPECT training
USC INFORMATION SCIENCES INSTITUTE
EXPECT
18
Results


The subject was able to add two new critiques of
both low and medium complexity.
The time taken was comparable to that for the other
four subjects that had previous training in Expect:
Time in
minutes
20
18
16
14
12
10
8
6
4
2
0
new user
Average of
other users
Completeness
critique
(simpler)
USC INFORMATION SCIENCES INSTITUTE
Resource
check critique
(more complex)
EXPECT
19
EXPECT: A User-Centered Framework for
Developing KBSs
EXPECT
Method
instantiator
Domain
dependent
KBS
KBS
compiler
Ontologies and
Method libraries
Knowledge Base
Domain
ontologies
and factual
knowledge
CYC/Sensus
Upper
Problem
solving
methods
Interdependency
Model (IM)
KA tools
EMeD
PSMTool
KA Strategies
KA Scripts
Knowledge-Based
System
USC INFORMATION SCIENCES INSTITUTE
COA
ontologies
Plans
(PLANET)
Evaluations and
Critiques
Resources
(OZONE)
Evaluation
PSMs
Instrumentation
EXPECT
20
EXPECT: A User Centered Approach for
Knowledge-Based Planning Tools
Knowledge acquisition technology that can guide users to
specify planning knowledge and develop planning tools

Expressive representations
– Loom/Powerloom KR&R
– EXPECT’s language to represent problem solving knowledge

Powerful reasoners
– Loom/Powerloom pattern classifier & reasoners
– abstract problem solving through partial evaluation


ex: how to move <a set of units> from a <location> to another <location>
Explicit models of planning knowledge and plan reasoners:
– PLANET ontology of plans, OZONE resource ontology
– plan evaluation and planning methods

Expectation-based knowledge acquisition tools
– Derive interdependencies between individual knowledge fragments
– KA Scripts to guide users in completing complex modifications
USC INFORMATION SCIENCES INSTITUTE
EXPECT
21