Continuing Work in Model-Based User Interfaces Jeffrey Nichols 05-830: Advanced User Interface Software.
Download
Report
Transcript Continuing Work in Model-Based User Interfaces Jeffrey Nichols 05-830: Advanced User Interface Software.
Continuing Work in
Model-Based User Interfaces
Jeffrey Nichols
05-830: Advanced User Interface Software
Last time…
Model-based User Interfaces
Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
Research continued…
Multiple models were integrated
Relational models were developed to explicitly link
other kinds of models
Data models became more detailed
Task models became very important
Research continued…
Fragmented into two different approaches
Software engineering approach (early 90’s-)
Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)
Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every
combination
What are task models, anyway?
Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure
Introduced by GOMS
Number of different task modeling languages
GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees
Developed by Fabio Paterno et al.
for the design of user interfaces
Goals
Graphical for easy interpretation
Concurrent model for representing UI
tasks
Different task types
Represent all tasks, including those
performed by the system
Used almost universally by new
model-based systems
Task Building Process
Three phases
Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.
Temporal Relationships
T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example
Note: First example is ambiguous
Another Example
Building/Editing Task Models
Tools are available
ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Software Engineering Approach
Lots and lots of systems!
MasterMind
Angel Puerta’s work at Stanford
UIML
Cameleon Project
Mecano, Mobi-D, XIML
Teresa
USIXML
etc…
MasterMind
One of the first system to integrate multiple models together
Mobi-D
Set of tools to support
a clearly defined
development cycle
Uses a series of
different models
Explicit relationships
that specify how the
models are related to
each other
Explicit interaction
between end users
and developers
Mobi-D Models
User-task Model
Domain Model
Describes the human-computer conversation at a low level
Presentation Model
Models of the entities that are manipulated in an interface and their
properties
Dialog Model
Describes tasks the user performs and what interaction capabilities are
needed
Specifies how the interaction objects appear in all of the different states
of the interface.
Relations
Describes how each of the models relate to each other
Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.
Mobi-D Process
1. Elicit user tasks
Start with informal description and then convert to outline (basis for task models in
Mobi-D)
More formal task analysis methods could probably be used
2. User-task and domain modeling
Skeleton domain model is built from task outline
Developer refines model
Explicit methods for generalizing pieces of model for reuse in other interface designs
3. Presentation and Dialog design
Decision support tools provide recommendations from model and interface guidelines
(if available)
System steps through task model and helps designer build final interface
By carrying the task model through the process, Mobi-D’s designers believe
users can provide more useful feedback
Mobi-D Discussion
Provides assistance rather than automating design
Recommendations do not limit flexibility, but
organize the design process
For usability engineers, not everyday users
Benefits come from reuse among small projects or for
managing interaction data from a large project
Models can be large and appear to require significant effort
to develop
Spawned a profitable company that does UI work
XIML
eXtensible Interface Markup Language
Based on Mobi-D work
Supports full development lifecycle
Used by RedWhale Software to drive
their interface consultant business
They have developed many tools
move interaction data to/from XIML
Leverage data in XIML to better
understand various interfaces
Automate parts of the interface design
process
Example Use for XIML
Multi-platform interface development
Other Software Engineering Systems
UIML
Originally a research project at Virginia Tech, now being developed commercially
by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa
(http://www.harmonia.com/)
(http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML
(http://www.usixml.org)
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very
complex)
Ubiquitous Computing Approach
“Pervasive computing cannot succeed if every device
must be accompanied by its own interactive software
and hardware…What is needed is a universal
interactive service protocol to which any compliant
interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is
interactively insufficient.
Ubiquitous Computing Approach
There are two problems here:
Infrastructure issues
How do devices communicate?
How do devices discover each other?
User Interfaces issues
Are devices described sufficiently to build a good UI?
How are interfaces generated?
How can one interface be created for controlling
combinations of related devices?
Infrastructure Issues
Possible to investigate these issues without
automatically generating UIs
Being addressed by lots of systems
Commercially
Research
UPnP, JINI, Salutation, HAVi
Speakeasy (PARC), many others…
Most systems that address the UI issues also have
some infrastructure component
Systems addressing UI issues
XWeb
ICrafter
A system for integrating user interfaces from multiple
devices
Supple
Now known as ICE – Interactive Computing Everywhere
A system for automatically generating interfaces with a
focus on customization/personalization.
Personal Universal Controller
My research…
XWeb
Work by Dan Olsen and group at BYU
Premise:
Apply the web metaphor to services in general
Support higher levels of interactivity
XWeb Protocols
Based upon the architecture of the web
XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects
xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP
CHANGE message (similar to GET in HTTP)
Sequence of editing operations to apply to a sub-tree
Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent Interfaces
Two models are specified
DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”
Atomic
Numeric
Time
Date
Enumeration
Text
Links
Aggregation
Group
List
XWeb Example
DataView
XWeb Example
XView
XWeb Example
Interface
Other XWeb Details
Has simple approach for
adjusting to different screen
sizes
Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces
Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter
Part of the Interactive Workspaces research project
at Stanford
Main objective:
“to allow users of interactive workspaces to flexibly interact
with services”
Contribution
An intelligent infrastructure to find services, aggregate them
into a single interface, and generate an interface for the
aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation accomplished?
High-level service interfaces (programmatic)
Data Producer
Data Consumer
The Interface Manager has pattern
generators
Recognize patterns in the services used
Generate interfaces for these patterns
This means that unique functionality will not be available
in the aggregate interface
Automatic Generation in ICrafter
Manual Generation in ICrafter
Supple
Eventual goal is to support automatic
personalization of user interfaces
Treats generation of interfaces as an
optimization problem
Can take into account usage patterns in
generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in
Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
Modeling Users with Traces
Supple uses traces to keep a usage model
Sequences of events:
<interface element, old value, new value>
Interfaces are rendered taking the traces into
account (though traces are not required)
Trails are segmented at interface close or reset
Generating with Optimization
Uses a branch-andbound search to
explore space of
alternatives
Guaranteed to find
an optimal solution
Optimization Equation
Cost of navigation between
subsequent controls in a trace
Device-specific measure
of how appropriate a
control is for manipulating
a variable of a given type
Total cost for one user trace
Total cost of all traces
Screenshots
Personal Universal Controller
My work with Brad
Problem:
Appliance interfaces are too complex and too
idiosyncratic.
Solution:
Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:
PDA, mobile phone, etc.
Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
The PUC System Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
Research Approach
1. Hand-design remote control interfaces
2. Determine functional information needed
from appliances to design user interfaces
3. Design language for describing appliance
functions
4. Build interface generators for multiple
platforms
5. Perform user studies to evaluate the
interface generators
Language Design
Informed by hand-designed interfaces
What functional information was
needed to create interfaces?
Additional Requirements
Support complete functionality of
appliance
No specific layout information
Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
Language Elements
Elements
State variables & commands
Labels
Group tree
Dependency information
Example media player
specification
Play, stop, pause, next track,
previous track
Play list
Language Elements, cont.
State Variables and
Commands
Represent functions of
appliance
State variables have types
Boolean, Enumeration,
Integer, String, etc.
Variables sufficient for most
functions but not all
e.g. “seek” button on a Radio
Language Elements, cont.
Label Information
One label not suitable everywhere
The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”
A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Label Information
One label not suitable everywhere
The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”
A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Group Tree
Specify organization
of functions
We use n-ary tree
with variables or
commands at leaves
Also used for
specifying complex
types
Lists
Unions
Language Elements, cont.
Group Tree
Specify organization
of functions
We use n-ary tree
with variables or
commands at leaves
Also used for
specifying complex
types
Lists
Unions
Language Elements, cont.
Dependency Information
Formulas that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less
Than, Is Defined
Linked with logical operators
(AND, OR)
Allows feedback to user when
a function is not available
Graphical Interface Generator
Rule-based approach
Multiple phases that
iteratively transform a
specification into a
user interface
Focuses on panel structure of
user interface
Small groups of controls have
basic layouts
Complexity comes from
structure of groups
Structure can be inferred
from dependency info!
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
Without layout fixing rules
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
With layout fixing rules
Controlling Appliances
We have built adaptors for many
actual appliances
Sony Digital Camcorder
Windows Media Player
Axis UPnP Pan & Tilt Camera
Lutron Lighting
X10 Lighting
Audiophase Shelf Stereo
AudioReQuest MP3 player
GM Vehicle Information System
GM Vehicle Climate Control
Written specifications for others
Elevator
Telephone/Answering Machine
GM Navigation System
Several Alarm Clocks