Transcript Kein Folientitel

IUI99, International Conference on Intelligent User Interfaces
Los Angeles, January 6th, 1999
Agent-based Multimedia Interaction
for Virtual Web Pages
Wolfgang Wahlster
German Research Center for Artificial
Intelligence, DFKI GmbH
Stuhlsatzenhausweg 3
66123 Saarbruecken, Germany
phone: (+49 681) 302-5252/4162
fax: (+49 681) 302-5341
e-mail: [email protected]
WWW:http://www.dfki.de/~wahlster
© W. Wahlster, DFKI
Outline
l What are Virtual Web Pages?
l What are Agent-Based User Interfaces?
l Using Life-like Characters for the Personalization of User Interfaces
l Plan-based Media Transformation and Coordination
l The Combination of Retrie ved and Generated Media Objects
for the Generation of Virtual Web Pages
l Commercial Applications in Advanced WebCommerce
l SmartKom: a Transportable and Transmutable Multimodal Interface
l Our Research Agenda for Agent-based User Interfaces
l Conclusion
© W. Wahlster, DFKI
Three Generations of Web Sites
First Generation
Second Generation
Interactive Web
Sites
Static Web Sites
Fossils cast in HTML
Third Generation
Virtual Web Sites
JavaScripts and Applets
Netbots,
Information Extraction,
Presentation Planners
Database Access and
Template-based Generation
User Modeling,
Machine Learning,
Online Layout
Dynamic Web Sites
Adaptive Web Sites
© W. Wahlster, DFKI
What is a Virtual Web Page?
Virtual Memory, Virtual Relation, Virtual Reality...
A Virtual Web Page
l is generated on the fly as a combination of various media objects from
multiple web sites or as a transformation of a real web page.
l looks like a real web page, but is not persistently stored.
l integrates generated and retrieved material in a coordinated way.
l can be tailored to a particular user profile and adapted to a particular
interaction context.
l has an underlying representation of the presentation context so that
an Interface Agent can comment, point to and explain its components.
© W. Wahlster, DFKI
Virtual Webpage Retrieved from 5 Different Servers
© W. Wahlster, DFKI
Virtual Webpage Augmented by Persona
© W. Wahlster, DFKI
What are Agent-based User Interfaces?
l appear as life-like
characters
l plan interactive
behavior
autonomously
l can initiate interaction
l respond immediately
to interruptions
l criticism and
clarification questions
l direct manipulation
INTERFACE
AGENTS
PROACTIVE
l anticipate the user's needs
l adopt the user's goals
l provide unsolicited comments
© W. Wahlster, DFKI
Intelligent Web Services
Consumer
buys
l Information
l Goods
l Services
Web Sites
Netbot
l Intelligent Parallel Retrieval
l Information Extraction and
Summarization
l Personalized Presentation
l Matchmaking
l Teleshopping Assistance
l Telemarketing Assistance
l Translation Services
l Data Mining Services
Provider
sells
l Information
l Goods
l Services
Knowledge about:
l Usage Patterns
l User Models
l Consumer Profiles
© W. Wahlster, DFKI
Netbots as Personal Assistants for WWW Users
Netbot := Intelligent Agent that uses Internet tools on a person’s behalf
e.g. Ahoy!,
Jango,
AiA
Netbots
Personal
Assistants
Indices,
Directories,
Search Engines
Mass
Services
WWW
Traveller’s Netbot: Tries to achieve traveller’s goals
(finding and executing plans)
l checks availability
l finds best price
l uses personal preferences (e.g. frequent flyer programme, seating preferences
l lets the traveller know, when seats become available (active help)
© W. Wahlster, DFKI
A Netbot for Portrait Photos: The Personal Picture Finder
http://finder.dfki.de:7000
Portrait Photo
<Name>
Netbot: Personal Picture Finder
Parallel MetaSearch of
Webpages for
<Name>
WWW
Parallel Search
in Picture
Archives
Home Pages
Extraction of
Images
Filtering of
Logos, Graphics,
...
Knowledge Sources
Applications: Journalism, Contact Preparation, Tracing Criminals
© W. Wahlster, DFKI
The Personal Picture Finder
© W. Wahlster, DFKI
Netbots versus Push Technologies
Push Technologies
1
Customer 1
Customer 2
Information
Provider
+
-
no effort for customer
+
-
good adaptation
+
+
good adaptation
minimal adaptation
Customer N
2
Interactive Pull
Query
Customer
Provider
Information
3
major effort for customer
Netbots with Parallel Pull
Provider 1
Query
Customer
Netbot
Provider 2
Information
minimal effort for customer
Provider N
© W. Wahlster, DFKI
Enhancement of User Interfaces through
Personalization
System is able to flexibly
tailor presentations to the
individual user and the
current situation.
An animated character
serves as “Alter Ego”
of the presentation system.
Personalized Presenters at DFKI
© W. Wahlster, DFKI
Personalized Package Presentation by an
Animated Agent
l Personalization =
adaption of system behavior
according to a user model
l Personalization =
the “agent” appears as
animated character which
presents, explains and
comments an offer, and to
which the user can talk to.
© W. Wahlster, DFKI
Video Character Presents the Interior of a Boeing 757
© W. Wahlster, DFKI
Reactive Behavior of the Persona Agent
© W. Wahlster, DFKI
PPP’s Persona Server implements a generic Presentation Agent
that can be easily adapted to various applications
Visual Appearances
Behaviors
l
l
l
l
Presentation Gestures
Reactive Behaviors
Idle-time actions
Navigation actions
Hand-drawn
Cartoon
Bitmaps
Persona
Server
Auditory Characteristics
l Sound effects, auditory icons
l Voice: male, female
Video Bitmaps
Generated
Bitmaps
from
3D-Models
© W. Wahlster, DFKI
The frames of the visual appearance of persona can be
cartoon-like images or video-frames which show real
persons
More than 200 cartoon frames were drawn by a professional artist.
A real persona was filmed with a video-camera and the pictures were
digitized with a frame grabber.
© W. Wahlster, DFKI
The Persona Editor
© W. Wahlster, DFKI
Context-Sensitive Decomposition of Persona Actions
High-Level
Persona Actions
Context-Sensitive
Expansion
take-position (t1 t2 )
point-to (t3 t4 )
move-to (t1 t2 )
r-stick-pointing (t3 t4 )
(including Navigation Actions)
Decomposition
into
Uninterruptable
Basic Postures
r-turn (t1 t21 )
r-hand-lift (t3 t31 )
r-step (t21 t22 )
f-turn (t22 t2 )
r-stick-expose (t31 t4 )
Bitmaps
...
...
...
...
© W. Wahlster, DFKI
PPP System Architecture
Multimedia
Generation
Text
Generator
Multimedia
Server
Production
Acts
Graphics
Generator
Gesture
Generator
Animation
Generator
Music
Generator
Generated
Material
Presentation
Planner
(PREPLAN)
KR &
Reasoning
(RAT)
Presentation
Acts
Layout
Manager
Signals +
Events
Persona
Server
© W. Wahlster, DFKI
Task of the Presentation Planner
Plan multimedia material as well as presentation acts and
their temporal coordination
Presentation Acts
Persona Acts
Display Acts
This is the
transformer
© W. Wahlster, DFKI
Persona explains a modem
© W. Wahlster, DFKI
Extensions of the Representation Formalism
Distinction between production and presentation acts
(i.e. Persona- or display acts)
Explicit representation of qualitative and quantitative constraints
Production Act
Presentation Act
Introduce
CreateGraphics
S-Show
S-Position
Elaborate-Parts
S-Wait
Label
Label
S-CreateWindow
S-Depict
S-Point
S-Speak
S-Point
S-Speak
Qualitative constraints: Create-Graphics meets S-Show, ...
Metric constraints: 1 <= Duration S-Wait <= 1, ...
© W. Wahlster, DFKI
Presentation Strategies in PPP contain qualitative and
metric constraints
(cf. [André/Rist 97])
(define-plan-operator
:HEADER (A0 (INTRODUCE P A ?object ?window))
:INFERIORS ((A1 (CREATE-GRAPHICS P A ?window ?object))
(A2 (S-SHOW P A ?window ?object))
(A3 (S-POSITION P A ?window))
(A4 (S-WAIT P A))
(A5 (ELABORATE-PARTS P A ?object ?window)))
:QUALITATIVE
((A1 (m) A2) (A3 (s) A2) (A3 (m) A5) (A5 (m) A4) (A4 (f) A2))
:METRIC
((10 <= DUR A2) (2 <= DUR A4 <= 2))
:START A1
:FINISH A2)
© W. Wahlster, DFKI
PPP first builds up a preliminary schedule at design time
PRELIMINARY SCHEDULE
© W. Wahlster, DFKI
The preliminary schedule is continously updated at
presentation time
UPDATED SCHEDULE
© W. Wahlster, DFKI
Temporal Reasoning in the Presentation Planner for
Dynamic Multimedia Coordination
Temporal
Consistency
Checker and
Propagator
Plan
Scheduler
Plan Nodes with
Links to Local
Temporal
Constraint
Networks
Plan Operators
with Metric and Qualitative
Temporal Constraints
© W. Wahlster, DFKI
Persona Presents an Automatically Designed
Business Chart
© W. Wahlster, DFKI
The Combination of Retrieved and Generated Media
Objects for Virtual Webpages
Multi-Domain
Problem Specs
Multiple
Data Sources
Information Structures
l Relations, Lists
l KR Terms
NETBOT
Retrieved
Results
Distributed
Information
Media Objects
l Texts, Sounds, Videos
l Pictures, Maps,
Animations
© W. Wahlster, DFKI
The Combination of Retrieved and Generated Media
Objects for Virtual Webpages
Information Structures
l Relations, Lists
l KR Terms
Select Canned
Media Objects
Design New
Media Objects
l Icons,
Clip Art
l Frames,
Sounds
l Graphics,
Animation
l Text,
Speech,
Mimic
Select & Design
Retrieved
Results
Media Objects
l Texts, Sounds, Videos
l Pictures, Maps,
Animations
Coordinate
Media Objects
l Temporal
Synchronization
l Spatial
Layout
Transform
Media Objects
l Clip, Convert,
Abstract
l Zoom, Pan,
Transition
Effects
Reuse & Transform
© W. Wahlster, DFKI
Virtual Webpage with Animation Effects Based on a
Single GIF Image
© W. Wahlster, DFKI
Transition Effects in a Series of Retrieved Pictures
© W. Wahlster, DFKI
Persona as a Personal Travel Consultant
© W. Wahlster, DFKI
The Generation of Virtual Webpages with PAN and AiA
Hotel
Agent
Trip Data
Address
Netbot
PAN
Map
Agent
AiA
Pictures and
Graphics
Presentation
Planner
Pieces of Text
Components
of virtual
Webpages
Coordinates for
Pointing Gestures
Input for Speech
Synthesis
Icons for
Hyperlinks
Virtual
Web
Presentation
Persona
Server
Constraintbased
Online
Layout
Weather
Agent
Train & Flight
Scheduling
Agent
Major Event
Agent
© W. Wahlster, DFKI
Persona as a Personal Travel Consultant
© W. Wahlster, DFKI
Dynamic Node Expansion for the Conditional
Generation of Virtual Web Pages
Introduce
Design-IntroPage
Elaborate
Emphasize
Location Link
Selected
S-Include- Illustrate S-Include- S-Speak
Text
Link
S-IncludePhoto
Location
S-IncludeMap
Label
S-Speak
This hotel has a
nice swimming
pool.
S-Point
Your hotel is
located here.
Default Time
Over/ Up
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Use of a Life-like Character for Electronic Commerce
© W. Wahlster, DFKI
Jennifer James as a Virtual Sales Agent
© extempo systems inc.
© W. Wahlster, DFKI
Sending Interface Agents to Clients: Plug-Ins or Applets?
Plug-Ins
l
l
l
l
l
Add features (character players) to browser
Download triggered by user
Requires disk space on client
Unrestricted access to client
Less appropriate for WebCommerce,
Guides
l Agents integrated in 3D environments
l Appropriate for Entertainment
Examples:
l Extempo's Jennifer James
(Hayes-Roth et al. 98)
l PFMagic's virtual petz
Applets
l
l
l
l
l
l
l
Java animation code sent over the net
Automatic loading
Requires no disk space on client
Restricted access to client
Appropriate for WebCommerce, Guides
Agents integrated in 2D environments
Less appropriate for Entertainment
Examples:
l DFKI's Web Persona
(Müller et al. 98)
l ISI's Adele (Johnson et al 98)
New in AiA/PAN: Balanced combination of Applets and Servelets
Efficient distribution of client-side Java and server-side Java for driving the
Interface Agent
© W. Wahlster, DFKI
Alternative Business Models for Shopbots
2
pays usage fee
3
pays fee after successful
commercial transaction
Provider
Internet
Shop 1
Queries
.
.
.
.
.
Shopbots
Transactions
Internet
Shop n
1
Banner fee
1
2
Usage fee
3
Transaction fee (credit card model)
pays fee for
banner avertisement
Provider
© W. Wahlster, DFKI
Intelligent Interface Technology is a Prerequisite
for Advanced WebCommerce
Virtual Web
Pages
Shopbots for
Automated
Comparison
Shopping
Text Analysis
and Generation
Multimedia
Presentation
Planning
Information Extraction from
HTML/XML Documents
Advanced
WebCommerce
User Modeling and Language Generation
Coordinated Text & Graphics Planning
Machine
Translation
Intuitive,
Multilingual
Access
Multimodal
Interfaces
Robust Dialogue
Understanding
Advanced Speech
Synthesis
One-to-One
Marketing
Dialogue with
Virtual Sales
Agents
© W. Wahlster, DFKI
SmartKom: A Transportable and Transmutable Interface Agent
Media Analysis
Media
Design
SmartKom-Mobile:
A Handheld
Communication
Assistant
Kernel of
SmartKom
Interface
Agent
Application
Management
Interaction Management
SmartKom-Public:
A Multimodal
Communication
Booth
SmartKom-Home/Office:
A Versatile Agent-based Interface
© W. Wahlster, DFKI
SmartKom: Intuitive Multimodal Interaction
Project Budget:
Project Duration:
$ 34 M
4 years
The SmartKom Consortium:
Main Contractor
Project Management
Testbed
Software Integration
DFKI
Saarbrücken
Uinv. Of
Munich
MediaInterface
Berkeley
Dresden
Saarbrücken
European Media Lab
Heidelberg
Univ. of
Erlangen
DAIMLERCHRYSLER
Aachen
Ulm
Univ. of
Stuttgart
Munich
Stuttgart
© W. Wahlster, DFKI
SmartKom-Public:
A Multimodal
Communication Booth
Loudspeaker
Room microphone
Smartcard/
Credit Card
for authentication
and billing
Face-tracking camera
Virtual touchscreen
protected against vandalism
Multipoint video conferencing
Docking station
for PDA/Notebook/
Camcorder
high speed and broad
bandwidth Internet
connectivity
High-resolution scanner
© W. Wahlster, DFKI
SmartKom-Mobile: A Handheld Communication Assistant
GPS
GSM for Telephone,
Fax, Internet
Connectivity
Camera
Wearable
Compute
Server
Stylus-Activated Sketch Pad
Microphone
MOBILE
Biosensor
for Authentication
& Emotional Feedback
Loudspeaker
Docking Station
for Car PC
© W. Wahlster, DFKI
SmartKom-Home/Office: A Versatile Agent-based Interface
SpeechMike
Natural Gesture Recognition
Virtual Touchscreen
© W. Wahlster, DFKI
The Architecture of the SmartKom Agent
Input
Processing
(cf. Maybury/Wahlster 1998)
Media
Media
Analysis
Analysis
Interaction
Management
Language
Media Fusion
Gesture
Discourse
Modeling
Biometrics
Media
Design
Design
Intention
Recognition
Language
User(s)
Graphics
User
Modeling
Application Interface
Graphics
Information
Applications
People
Gesture
Animated
Presentation
Agent
Presentation
Design
Output
Rendering
User
Model
Discourse
Model
Domain
Model
Task
Model
Media
Models
Representation and Inference
© W. Wahlster, DFKI
Our Research Agenda for Agent-based Interfaces
(Wahlster, André, Rist, Müller, Graf etc; www.dfki.de/imedia)
1
2
3
Personalized Presentation Agents
(limited user interaction)
Personalized Interface Agents
(full user interaction)
WIP: 1989-1993
PPP:1994-1996
Multiple Interface Agents
(agent-agent and user interaction)
l Multiple Presentation Agents in one scene
(eg. pros and cons)
l Multiple Role-Taking (eg. Travel Assistance vs.
Comparison Shopper)
l Multiple Interface Agents (eg. human-computer,
human-human interaction)
AiA: 1997-2000
© W. Wahlster, DFKI
Multiple Agents Discussing Pros and Cons of a Mercedes
Model
I recommend you this
SLX limousine.
© W. Wahlster, DFKI
Research on Intelligent Web Services brings disparate
subfields in the area of intelligent systems together
Planning
User
Modeling
Knowledge
Representation
Image
Understanding
Intelligent
Intelligent
Web
Natural Language
Web
Services
Understanding
Services
Machine
Learning
Plan
Recognition
Information
Multimodal Retrieval
User
Interfaces
© W. Wahlster, DFKI
Conclusion
ECommerce projects of DFKI have shown that research on
agent-based multimodal interfaces can be transferred to real
real-world applications:
Dekra (largest European organization of used car dealers):
FairCar as an ECommerce platform with NL access and a
comparison shopping agent for used cars
DaimlerChrysler: IKP for online user modelling in a one-toone marketing system for Mercedes cars
Otto/Shopping24/Eddie Bauer (largest European mail order
company): virtual sales agents for one-to-one marketing of
fashion and computer hardware
Lufthansa/Condor: direct marketing of charter flights
© W. Wahlster, DFKI
Conclusion
The generation of virtual web pages by agent-based
multimodal interfaces leads to innovative applications in:
Electronic Commerce, Electronic TV Guides (EPG)
Telelearning environments, Call Centers and Help Desks
Two Research Challenges:
Making the interface agents sensitive to temporary
limitations of the user´s time and working memory
capacity (cf. our READY project, Jameson et al., p. 79-85
in IUI99 Proceedings)
Making the agents instructible, so that they can learn
from the user in a dialog, how to extract information in a
new domain (cf. our PAN project, Bauer/Dengler, p. 153156 in IUI99 Proceedings)
© W. Wahlster, DFKI