Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software.
Download
Report
Transcript Recent Work in Model-Based User Interfaces Jeffrey Nichols Lecture #13 05-830: Advanced User Interface Software.
Recent Work in Model-Based User Interfaces
Jeffrey Nichols
Lecture #13
05-830: Advanced User Interface Software
Last time…
Model-based User Interfaces
Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
But work continued…
Focus Changed
Task models were leveraged more
Design assistant aspect emphasized
A Couple Projects of Interest:
Trident
Mecano & Mobi-D
FUSE
AIDE
TRIDENT
Vanderdonckt, J., Knowledge-Based Systems for Automated User Interface Generation: the
TRIDENT Experience, Technical Report RP-95-010, Facultes Universitaires Notre-Dame de la
Paix, Institut d’Informatique, Namur, 1995.
An interface design assistant
Interesting features:
Knowledge-based approach (i.e. expert system)
Choosing Widgets
Doing Layout
Use of Task Models
Decides where separate windows are needed
Choosing Widgets
Used a decision tree
Chose abstract
interaction objects
(AIO)
Similar to Brad’s
Interactor Model
Lots of parameters
Continuous?
Capacity
Etc.
Choosing Layout
Uses Right/Bottom Strategy
Next component is placed to the right or below
the current component
Decision made by heuristics or designer
Right/Bottom Strategy
Windows from Task Models
Basically used for constructing wizard-like interfaces
What information should be on the first screen, etc.
What are task models, anyway?
Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure
Introduced by GOMS
Number of different task modeling languages
GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees
Developed by Fabio Paterno et
al. for the design of user
interfaces
Goals
Graphical for easy interpretation
Concurrent model for
representing UI tasks
Different task types
Represent all tasks, including
those performed by the system
Task Building Process
Three phases
Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.
Temporal Relationships
T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example
Note: First example is ambiguous
Another Example
Building/Editing Task Models
Tools are available
ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Recent Systems
XIML – eXtensible Interface Markup Language
XWeb
Now known as ICE – Interactive Computing Everywhere
ICrafter
Developed by the makers of Mecano/Mobi-D and Trident
Kitchen-sink language for modeling any part of the
interface design process
A system for integrating user interfaces from multiple
devices
Personal Universal Controller
My research…
XIML
eXtensible Interface Markup Language
Designed by RedWhale Software
Intended to support the
full lifecycle of interface
building
XIML Requirements
Central Repository of Data
For one user interface or many
Comprehensive Lifecycle Support
Abstract and Concrete Elements
Relational Support
Underlying Technology
XIML must be independent of particular tools
Models in XIML
An XIML document can contain any type of
model
Task
Domain
User
Presentation
Dialog
Example Use for XIML
Multi-platform interface development
Status of XIML
Used by RedWhale Software to drive their
interface consultant business
They have developed many tools
move interaction data to/from XIML
Leverage data in XIML to better understand various
interfaces
Automate parts of the interface design process
Model-Based Interfaces for Control
XWeb
ICrafter
PUC
XWeb
Work by Dan Olsen and group at BYU
Premise:
“Pervasive computing cannot succeed if every device must
be accompanied by its own interactive software and
hardware…What is needed is a universal interactive
service protocol to which any compliant interactive client
can connect and access any service.”
The web comes close to solving this problem, but is
interactively insufficient.
XWeb Protocols
Based upon the architecture of the web
XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects
xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP
CHANGE message (similar to GET in HTTP)
Sequence of editing operations to apply to a sub-tree
Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent Interfaces
Two models are specified
DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”
Interactors are somewhat like abstract interaction objects
Atomic
Numeric
Time
Date
Enumeration
Text
Links
Aggregation
Group
List
XWeb Example
DataView
XWeb Example
XView
XWeb Example
Interface
Other XWeb Details
Has simple approach for
adjusting to different screen
sizes
Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces
Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter
Part of the Interactive Workspaces research project
at Stanford
Main objective:
“to allow users of interactive workspaces to flexibly interact
with services”
Contribution
An intelligent infrastructure to find services, aggregate them
into a single interface, and generate an interface for the
aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation accomplished?
High-level service interfaces (programmatic)
Data Producer
Data Consumer
The Interface Manager has pattern
generators
Recognize patterns in the services used
Generate interfaces for these patterns
This means that unique functionality will not be available
in the aggregate interface
Automatic Generation in ICrafter
Manual Generation in ICrafter
Personal Universal Controller
My work with Brad
Problem:
Appliance interfaces are too complex and too
idiosyncratic.
Solution:
Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:
PDA, mobile phone, etc.
Idea
Specifications
Control
Feedback
Control existing appliances
Generate multi-modal interfaces
Architecture - Appliance
Comm. Protocol
Interface
Specification
Generators
Adaptors
Lang.
XML-based
Language Design
Approach
Create reference interfaces
Test interfaces with subjects
AIWA Shelf Stereo
AT&T Telephone/Answering
Machine
Users twice as fast and made half
the errors with reference interfaces
as compared to manufacturers’
interfaces
Analyze interfaces for functional
information
Language Elements
State Variables and Commands
Represent functions of appliance
State variables have types
Boolean, Enumeration, Integer, String, etc.
Variables sufficient for most functions but not all
“seek” button on a Radio
Label Information
One label not suitable everywhere
The optimal label length changes with screen size
Speech interfaces may benefit from pronunciation and
text-to-speech information
Language Elements, cont.
Group Tree
Specify organization
of functions
We use n-ary tree
with variables or
commands at
leaves
Language Elements, cont.
Dependency Information
Formulas that specify when a variable or
command is active in terms of other state
variables
Equals, Greater Than, Less Than
Linked with logical operators (AND, OR)
For example,
<and>
<equals state=“PowerState”>true</equals>
<equals state=“RadioBand”>AM</equals>
</and>
Interface Generators
Generators for Two Modalities
Graphical
Implemented for PocketPC in Java 1.1
Uses dependency information to generate panel structure of
interface
Speech
Implemented using Universal Speech Interface (USI)
techniques [Rosenfeld 2001]
Uses dependency information to disambiguate shortcut
words (e.g. “play”) and resolve pre-conditions for a
requested function (e.g. “play CD”)
Graphical Interface Generator
Focuses on panel structure
of user interface
Small groups of controls
have basic layouts
Complexity comes from
structure of groups
Structure can be inferred
from dependency info!
Inferring Structure
Find sets of variables that are
“mutually exclusive”
Every variable in a set will
never be active at the same
time as a variable in another set
Create structure with sets, using
overlapping panels
Choosing Panel Types
a)
b)
c)
full screen
tabbed
partial screen
Making the Interface Concrete
Finish conceptual layout
Choose controls (decision
tree)
Choose row layouts
(one column, two column, etc.)
Allocate space
Examine panel contents and
choose sizes
Instantiate and place controls
Generating Speech Interfaces
Automatically build USI tree from dependencies
Allows verbal navigation of functional groups
Automatically generate grammar for parser
Phrases for query and control
“What is playmode?”
“Set playmode to play”
“play”
Automatically generate language model and
pronunciation for recognizer