Transcript Dagstuhl (Agents, SAB)

Dagstuhl Seminar Semantics for the Web
Dagstuhl, 22 March, 2000
Generating Virtual Webpages
Wolfgang Wahlster
German Research Center for Artificial
Intelligence, DFKI GmbH
Stuhlsatzenhausweg 3
66123 Saarbruecken, Germany
phone: (+49 681) 302-5252/4162
fax: (+49 681) 302-5341
e-mail: [email protected]
WWW:http://www.dfki.de/~wahlster
© W. Wahlster, DFKI
Three Generations of Web Sites
First Generation
Second Generation
Interactive Web
Sites
Static Web Sites
Fossils cast in HTML
Third Generation
Virtual Webpages
JavaScripts and Applets
Netbots,
Information Extraction,
Presentation Planners
Database Access and
Template-based Generation
User Modeling,
Machine Learning,
Online Layout
Dynamic Web Sites
Adaptive Web Sites
© W. Wahlster, DFKI
What is a Virtual Web Page?
Virtual Memory, Virtual Relation, Virtual Reality...
A Virtual Web Page
l is generated on the fly as a combination of various media objects from
multiple web sites or as a transformation of a real web page.
l looks like a real web page, but is not persistently stored.
l integrates generated and retrieved material in a coordinated way.
l can be tailored to a particular user profile and adapted to a particular
interaction context.
l has an underlying representation of the presentation context so that
an Interface Agent can comment, point to and explain its components.
© W. Wahlster, DFKI
AiA: Information Integration for Virtual Webpages
PAN Travel Agent Andi
Car Route Planner
Yahoo
News
Server
Yahoo
Weather
Server
Hotel
Guide
Gault Millau
Restaurant
Guide
© W. Wahlster, DFKI
Virtual Webpage Retrieved from 5 Different Servers
© W. Wahlster, DFKI
The Combination of Retrieved and Generated Media
Objects for Virtual Webpages
Multi-Domain
Problem Specs
Multiple
Data Sources
Information Structures
l Relations, Lists
l KR Terms
NETBOT
Retrieved
Results
Distributed
Information
Media Objects
l Texts, Sounds, Videos
l Pictures, Maps,
Animations
© W. Wahlster, DFKI
The Combination of Retrieved and Generated Media
Objects for Virtual Webpages
Information Structures
l Relations, Lists
l KR Terms
Select Canned
Media Objects
Design New
Media Objects
l Icons,
Clip Art
l Frames,
Sounds
l Graphics,
Animation
l Text,
Speech,
Mimic
Select & Design
Retrieved
Results
Media Objects
l Texts, Sounds, Videos
l Pictures, Maps,
Animations
Coordinate
Media Objects
l Temporal
Synchronization
l Spatial
Layout
Transform
Media Objects
l Clip, Convert,
Abstract
l Zoom, Pan,
Transition
Effects
Reuse & Transform
© W. Wahlster, DFKI
Operational Models of Referential Semantics for
Robots and Netbots
Robot
Netbot
Set of Subsumption
Relations in an Ontology
Set of Subsumption
Relations in an Ontology
“Screw”
“Departure Time”
Set of Recognizers
Physical
Objects
Screw 1
...
Screw N
Set of Wrappers
WWW
Objects
DT 1
...
DT N
© W. Wahlster, DFKI
Enhancement of User Interfaces through Personalization
System is able to flexibly
tailor presentations to the
individual user and the
current situation.
An animated character
serves as “Alter Ego”
of the presentation system.
Personalized Presenters at DFKI
© W. Wahlster, DFKI
The Role of Ontological Annotations for the
Generation and Analysis of virtual Webpage
Webpages
with Ontological
Annotations
Webpages
without Ontological
Annotations
Information
Extraction Agent
Presentation
Planner
Virtual Webpage
Presentation
Agent
Persona
With Ontological Annotations in: SHOE, OML,
XOL, OIL
and Persona Annotation in
PML
Information
Extraction
Agents
TriAS
© W. Wahlster, DFKI
Video Dresdner Bank
© W. Wahlster, DFKI
The Generation of Virtual Webpages with PAN and AiA
Hotel
Agent
Trip Data
Address
Netbot
PAN
Map
Agent
AiA
Pictures and
Graphics
Presentation
Planner
Pieces of Text
Components
of virtual
Webpages
Coordinates for
Pointing Gestures
Input for Speech
Synthesis
Icons for
Hyperlinks
Virtual
Web
Presentation
Persona
Server
Constraintbased
Online
Layout
Weather
Agent
Train & Flight
Scheduling
Agent
Major Event
Agent
© W. Wahlster, DFKI
Persona as a Personal Travel Consultant
© W. Wahlster, DFKI
AIA’s presentation planner has been extended to
accommodate for various target platforms through the
introduction of a mark-up language layer
Presentation Planner
PETPML
PET
Persona
Player
Agent
Script
MS-Agent
Controller
SMIL
SMIL
Player
WML
WMLBrowser
© W. Wahlster, DFKI
PPP’s Persona Server implements a generic Presentation Agent
that can be easily adapted to various applications
Visual Appearances
Behaviors
l
l
l
l
Presentation Gestures
Reactive Behaviors
Idle-time actions
Navigation actions
Hand-drawn
Cartoon
Bitmaps
Persona
Server
Auditory Characteristics
l Sound effects, auditory icons
l Voice: male, female
Video Bitmaps
Generated
Bitmaps
from
3D-Models
© W. Wahlster, DFKI
Classification of Persona Gestures
Gesture Catalogue
Talking Posture 1
• cautious, hesitant
• appeal for compliance
• avoids body-gestures
Talking Posture 2
• active, attentive
• self-confident
• uses body-gestures
© W. Wahlster, DFKI
Context-Sensitive Decomposition of Persona Actions
High-Level
Persona Actions
Context-Sensitive
Expansion
take-position (t1 t2 )
point-to (t3 t4 )
move-to (t1 t2 )
r-stick-pointing (t3 t4 )
(including Navigation Actions)
Decomposition
into
Uninterruptable
Basic Postures
r-turn (t1 t21 )
r-hand-lift (t3 t31 )
r-step (t21 t22 )
f-turn (t22 t2 )
r-stick-expose (t31 t4 )
Bitmaps
...
...
...
...
© W. Wahlster, DFKI
Extensions of the Representation Formalism
Distinction between production and presentation acts
(i.e. Persona- or display acts)
Explicit representation of qualitative and quantitative constraints
Production Act
Presentation Act
Introduce
CreateGraphics
S-Show
S-Position
Elaborate-Parts
S-Wait
Label
Label
S-CreateWindow
S-Depict
S-Point
S-Speak
S-Point
S-Speak
Qualitative constraints: Create-Graphics meets S-Show, ...
Metric constraints: 1 <= Duration S-Wait <= 1, ...
© W. Wahlster, DFKI
Persona Presents an Automatically Designed
Business Chart
© W. Wahlster, DFKI
Virtual Webpage with Animation Effects Based on a
Single GIF Image
© W. Wahlster, DFKI
Transition Effects in a Series of Retrieved Pictures
© W. Wahlster, DFKI
PET: Persona-Enabling Toolkit
Objective:
l Enable non-professional computer users to populate their
web pages with lifelike characters
PET comes with:
l a set of characters and basic gestures
l an easy-to-learn Persona markup language
Developer’s PET will include:
l a character design tool which enables users to build their
own characters
Technical Realization:
l Based on XML and Java
© W. Wahlster, DFKI
The Persona Markup Language
Specification of
Persona actions
<html>
<head>
Specification of the
<title> Persona Test </title>
character to be used
</head>
<body>
<persona bitmap=“cartoon” ...>
<uselib url= .../>
<do name=“greet”/>
<do name =“speak” args=“hello”/>
</persona>
</body>
</html>
Features:
– XML-based
– easy to learn
© W. Wahlster, DFKI
Tasks of PET
PET-Parser
l Preprocessing (required for non-standard HTML pages)
l Identification of Persona Tags
PET-Generator
l Replacement of Persona Tag with Java Applet
l Determination of parameters for Java Applet
l Realization of Persona active elements as JavaScript
l Generation of DHTML layers (transparency of Applet box)
l Generation of data, e.g. audio files for speech output
l Generation of scripts
© W. Wahlster, DFKI
Functional View of PET
Bitmaps
Webpage with Reference to
Java Applet
URL of Webpage
with Persona Tag
<html>
<head>
<title> Persona Test </title>
</head>
<body>
<persona bitmap=“cartoon” ...>
<uselib url= .../>
<do name=“greet”/>
<do name=“standard”/>
<do name =“speak”
args=“hello”/>
</persona>
</body>
</html>
PET Application
Server
PET
Parser
PET
Generator
<html> ...
<APPLET
archive=“personaplayer.jar”...
</APPLET>
...</html>
Persona Scripts
waitscreen 4
gesture greet 0 0 null
gesture laugh 0 0 null
...
Persona Engine
Behavior Monitor
Audio Data
Event
Handler
Character
Composer
© W. Wahlster, DFKI
The Bidirectional Control Flow on
Persona-Enabled Webpages
Triggers
actions of the Persona
l Text Input
l Speech Input
l Menu Input
l Direct Manipulation
Input
Web Persona
l Mouse Clicks
l Mouse Movements
Triggers
operations on elements
of the webpage
© W. Wahlster, DFKI
Video 911 & Boxter
© W. Wahlster, DFKI
Persona Active Elements (PAE)
l
Active Images
An active image starts a persona action when clicked.
<ACTIVEIMAGE SRC=“image” HREF=“url” NAME=“image name”
STATUS=“status message” ALT=“tooltip” CACTION=“persona action onClick”
OACTION=“persona action on MouseOver” ...>
l
Addressable Objects
An addressable object is an object which can be addressed and
manipulated by Persona via its name and its position.
<PDIV DIVNAME=“name of the element” DVFRAME=“frame name”
TOP=“anchor-y” LEFT=“anchor-x”>some HTML elements</PDIV>
© W. Wahlster, DFKI
Spatial and Temporal Coordination of
Multimedia Presentations
• Multimedia coordination in previous systems:
– no declarative representation of spatial and temporal
layout
– no synchronization of Persona actions with other
dynamic multimedia objects
• Solution: Use PrePlan for automated generation of
SMIL expressions
– Synchronized Multimedia Integration Language
– officially recommended by W3 consortium
© W. Wahlster, DFKI
Approach
• High-level specification of temporal and spatial
constraints within the operators of the Presentation
Planner PrePlan
• Use of the incremental constraint solving toolkit
Cassowary
– Uniform treatment of spatial and temporal constraints
– Transformation of qualitative constraints into metric
constraints
– Integration of a backtracking mechanism to handle
disjunctions
© W. Wahlster, DFKI
Processing Steps
Decomposition of presentation goals into elementary acts
and collection of spatial and temporal constraints
Constraint propagation using extended Cassowary constraint
solver
Representation of spatial and temporal layout as SMIL
constructs
© W. Wahlster, DFKI
Example of a Plan Operator
(define-plan-operator
:header (A0 (ShowPresentation ?topic))
:constraints
(*and* (BELP (Illustrates ?video ?topic)) (BELP (Video ?video))
(BELP (Sets-to-Music ?audio ?topic)) (BELP (Audio ?audio))
(BELP (Summarizes ?title ?topic)))
:inferiors
(A1 (SAddSmilCode (?video)))
(A2 (SAddSmilCode (?audio)))
(A3 (SAddSmilCode (?title)))
:temporal ((A1 (d) A3) (2 <= begin A1 - begin A2))
:spatial
((aligntop A1) (centerh A1) (centerh A3)
(20 <= top A1 - bottom A3 <= 20))
© W. Wahlster, DFKI
Representation of Spatial and Temporal
Constraints
• Spatial constraints
– Qualitative constraints
(Image1 LeftOf Image2), (Image1 TopAlign Image2)
– Quantitative constraints
(4 < Top Image1 - Top Image2)
• Temporal constraints
– Qualitative constraints
Allen constraints: (meets, before, overlaps, ...)
– Quantitative constraints
Linear inequalities: (Begin Audio1 - End Audio2 < 3),
(2 <= Duration Audio1 <= 6)
© W. Wahlster, DFKI
Resulting SMIL-Document
<smil>
<head>
<meta content="Smiling Saarland" name="title"/>
<layout>
<root-layout height="50" width="300"/>
<region height="30" id="reg43463" left="0" top="0" width="300"/>
</layout>
</head>
<body>
<par>
<audio begin="1.0s" end="21.807s" src="snowman.wav"/>
<textstream begin="0.0s" end="22.807s" region="reg43463"
src="snowman.rt"/>
<video begin="3.0s" end="21.807s" src="snowman.avi"/>
</par>
</body>
</smil>
© W. Wahlster, DFKI
Interactive Web Demo
© W. Wahlster, DFKI
Synchronization of Persona Actions with other
Media Objects
Some Examples:
• point to an object when it appears in a video
• comment on a video 3 seconds after it has started
• Smile when video appears on the screen
• repeat lip movements until audio stops
© W. Wahlster, DFKI
Synchronization of Persona Actions
(define-plan-operator
:header (A0 (ShowPresentation ?topic))
:constraints
(*and*
(BELP (Describes ?audio ?topic)) (BELP (Audio ?audio))
(BELP (TalkingGesture ?video)) (BELP (RepeatGesture ?video))
:inferiors
(A1 (PresentPictureSequence (?topic)))
(A2 (SAddSmileCode (?audio)))
(A3 (SAddSmilCode (?video)))
:temporal ((A2 (d) A1) (2 <= begin A2 - begin A1) (A2 (e) A3))
:spatial
( (aligntop A1) (alignleft A1)
(1 <= bottom A0 - bottom A3 <= 1)
(1 <= right A0 - right A3 <= 1))
© W. Wahlster, DFKI
SMIL Specification for Persona Presentation
<smil> <head>
<meta name="title" content="Cyberella in RealPix Demo" />
<layout>
<root-layout width="650" height="385" background-color="#9898F8"/>
<region id=“reg471101" top="0" left="12" width="385" height="284"
background-color="#9898F8"/>
<region id=“reg081591" top="95" left=“354" width="295" height="385"
background-color="#9898F8"/>
</layout> </head> <body>
<par>
<img region= " reg471101" begin= " 0.0" src= "schwarzenacker.rp"
fill="freeze"/>
<animation begin= "2.0s" end="15.1" region="reg471101"
src= "talking-gst.rp"/>
<audio begin= "2.0s" end="15.1" src="schwarzenacker.rm" />
</par> </body> </smil>
© W. Wahlster, DFKI
Resulting Timeline Diagram
...
...
...
At archeological finds in Schwarzenacker, ...
...
...
© W. Wahlster, DFKI
Using SMIL to Synchronize Persona Actions
© W. Wahlster, DFKI
Video Otto 1
© W. Wahlster, DFKI
Information Extraction Agents
• Information Filtering
• Information Retrieval
• Information Integration
identify relevant
documents
– ...
– identify and extract relevant
pieces of information
– transform them into canonical form
wrappers
wrappers
• operational descriptions of a target concept
• abstract from concrete occurrence within document
• robust against modifications
© W. Wahlster, DFKI
The Trainable Information Agents Framework
combination of "classical"
problem-solving methods
and information agents
specifications
results
Application
info requests
User
preferences/heuristics
info
domain ontology
requests
training
planning knowledge
user preferences
Browser
InfoBroker
improved dialog guidance
info requests
PBD dialog
info or script
query
planning, optimization,
Web site annotations
and execution
Info Extraction
Trainer
© W. Wahlster, DFKI
High Degree of Parallelism of Queries
© W. Wahlster, DFKI
Knowledge about a Webpage Shared by
User and Agent
common part (usable
for communication)
structural
visual/semantic
Naive
User
procedural
Learning Annotation
Agent
© W. Wahlster, DFKI
Video PAN
© W. Wahlster, DFKI
Example - Ontology
Train_Connection [ from =>> Location;
to =>> Location;
travel_date =>> Date;
time =>> Time;
depart_time =>> Time;
arrive_time =>> Time;
cost =>> Price;
travel_duration =>> Duration;
info_url =>> URL;
... ]
© W. Wahlster, DFKI
Query Planning - I
• states: information states
– concepts / attributes and instantiations
• operators: querying schemes
– preconditions (´+´) and effects (´-´)
to time arrive_time travel_duration
< +, +, +, +, -, -, -, u, - >
from travel_date depart_time cost info_url
© W. Wahlster, DFKI
Query Planning - II
• merging of alternative plans with identical
intermediate states
• (a)
op2
• (a,b)
•
op3
op1
(a,c)
op4
•
(a,b,c,d)
•
•
(a,b,c,d,e)
•
(a,b)
op3
(a,b,c,d,f)
• (a,c)
op4
• (a,b,c,d)
(a,b,c,d)
op6
op5
merging
•
op1
• (a)op2
op5
op6
• (a,b,c,d,e) • (a,b,c,d,f)
© W. Wahlster, DFKI
Query Planning - III
Top
State
Ontology
City.value = München
City.language = German
...
String
...
Operators
CityName1
Language
City
value
–
–
+
+
< String, Language, String, Language >
babelfish
.
.
.
language
" c  pre (op ) $ i : S 0 = i :: c   c
c  S 0 = Int ( op )
© W. Wahlster, DFKI
Query Plan Visualization
• Features
– alternative queries
– past states
– future states
– state descriptions
– PBD requests
– accept / reject PBD
request
– assessment of plans
– expected completion time
© W. Wahlster, DFKI
Using the presentation planner to serve
mobile users
The new challenge
accommodate for device-specific display and
interaction limitations (e.g. 60*90 pixel displays)
Approach
- develop new designs for presenting information
on mobile devices
- identify elementary design elements
- define new presentation strategies for the
automated composition of designs
© W. Wahlster, DFKI
Application example: Route Descriptions over
the Mobile Phone
Planner
a selected path gets
translated by the planner
into a sequence of WML
pages to be displayed on a
mobile phone
“vertical
bar view”
“runway view”
“isometric
view”
© W. Wahlster, DFKI
Video Otto 2
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
Digital Assistant Selector
© W. Wahlster, DFKI
Simulated Dialogues as a Novel Presentation
Technique
• Presentation teams convey certain rhetorical
relationships in a more canonical way
– Provide pros and cons
• The single presenters can serve as indices which help
the user to classify information.
– Provide information from different points of view, e.g.
businessman versus tourist
• Presentation teams can serve as rhetorical devices that
allow for a continuous reinforcement of beliefs
– involve pseudo-experts to increase evidence
© W. Wahlster, DFKI
Presentation Teams
I recommend you this SLX
limousine.
© W. Wahlster, DFKI
Underlying Knowledge Base
• Representation of domain
– FACT attribute car_1 consumption_car_1
• Value dimensions for cars adopted from a study of
the German car market
– safety, economy, comfort, sportiness, prestige, family
and environmental friendliness
– FACT polarity consumption_car_1 economy negative
• Difficulty to infer implication of dimension on attribute
– FACT difficulty consumption_car_1 economy low
© W. Wahlster, DFKI
Example of a Dialogue Strategy
Question:
How much gas does it consume?
Answer:
It consumes 8l per 100 km.
Negative Response:
I’m worrying about the running
costs.
Dampening Counter:
Forget about the costs.
Think of the prestige!
Header:
(dampening_counter ?agent ?prop
?dim)
Constraints:
(*and*
(positive ?agent)
(pol ?prop ?other_dim
positive))
Inferiors:
(Speak ?agent
(“Forget about the ” ?dim “!”))
(Speak ?agent
(“Think of the ” ?other_dim “!”))
© W. Wahlster, DFKI
Current and Future Work: Multiple Interface Agents for
User-adaptive Decision Support
User-Adaptive Search Planning
...
...
weighted propositions
Multiple Decision Support Agents
Spare parts for this car
are rather expensive!
But, it’s
fast!
© W. Wahlster, DFKI
Conclusion
The generation of virtual webpages is heavily based on ontological
annotation
The plan-based approach to generating virtual webpages guarantees
high-level ontological annotations
Ontological annotations are needed not only for information
extraction agents but also for presentation agents
Realization procedures and wrappers form an important part of
the referential semantics of objects on the web
Using presentation planning with temporal and spatial constraint
processing the low-level media synchronization can be done
automatically using SMIL annotations
see www.dfki.de/~wahlster/semanticweb
© W. Wahlster, DFKI
The WebPersona and Virtual Webpages
Team at DFKI
Computer Science:
Graphics Design:
Elisabeth André
Mathias Bauer
Dietmar Dengler
Alexander Kröner
Marcus Meyer
Jochen Müller
Stephan Neurohr
Gaby Paul
Thomas Rist
Wolfgang Wahlster
Bernhard Kirsch
Renato Orsini
Peter Rist
Cognitive Psychology:
Susanne van Mulken
© W. Wahlster, DFKI