Lecture 13: Continuing Work in Model-Based User Interfaces Brad Myers Slides originally authored by Jeffrey Nichols, 2004 05-830: Advanced User Interface Software.
Download
Report
Transcript Lecture 13: Continuing Work in Model-Based User Interfaces Brad Myers Slides originally authored by Jeffrey Nichols, 2004 05-830: Advanced User Interface Software.
Lecture 13:
Continuing Work in
Model-Based User Interfaces
Brad Myers
Slides originally authored by Jeffrey Nichols, 2004
05-830: Advanced User Interface Software
Last time…
Model-based User Interfaces
Automatic generation of the user interface so the
programmer won’t do a bad job.
Dialog boxes are relatively easy to generate
The full application interface is hard to generate
Abstract descriptions of the interface can be
longer and harder to generate than implementing
the interface itself.
Interface builders turned out to be easier…
Research continued…
Multiple models were integrated
Relational models were developed to explicitly link
other kinds of models
Data models became more detailed
Task models became very important
Research continued…
Fragmented into two different approaches
Software engineering approach (early 90’s-)
Very detailed models to improve overall design process
Intelligent design assistants instead of automatic generation
Significant use of task models
Ubiquitous computing approach (2000-)
Tons of “invisible” processors that perform tasks for us
UIs for these processors are presented on other devices
(mobile phone, PDA, speech, etc.)
Impossible to manually build user interfaces for every
combination
What are task models, anyway?
Key part of many current HCI approaches
Description of the process a user takes to reach a
goal in a specific domain
Typically have hierarchical structure
Introduced by GOMS
Number of different task modeling languages
GOMS
UAN
ConcurTaskTrees
ConcurTaskTrees
Developed by Fabio Paterno et al.
for the design of user interfaces
Goals
Graphical for easy interpretation
Concurrent model for representing UI
tasks
Different task types
Represent all tasks, including those
performed by the system
Used almost universally by new
model-based systems
Task Building Process
Three phases
Hierarchically decompose
the tasks
Identify the temporal
relationships among tasks
at same level
Identify what objects are
manipulated and what
actions can be performed
on them, and assign these
to the tasks as appropriate.
Temporal Relationships
T1 [] T2 - Choice
T1 ||| T2 - Interleaving
T1 |[]| T2 - Synchronization
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 - Deactivation
T1* - Iteration
T1(n) - Finite Iteration
[T1] - Optional
T – Recursion
Example
Note: First example is ambiguous
T1 [] T2 - Choice
T1 ||| T2 - Interleaving
Another Example
T1 [] T2 – Choice
T1 >> T2 - Enabling
T1 []>> T2 - Enabling with
Information Passing
T1 [> T2 – Deactivation
Building/Editing Task Models
Tools are available
ConcurTaskTrees
Environment
http://giove.cnuce.cnr.it/ctte.html or Google for “ConcurTaskTrees”
Software Engineering Approach
Lots and lots of systems!
Commercial work
MasterMind
Angel Puerta’s work at Stanford
UIML
Cameleon Project
Mecano, Mobi-D, XIML
Teresa
USIXML
etc…
Software Engineering Approach
Commercial work
“Model-based design”
Example: BPMN “business
process modeling notation”
Business experts should be
able to author models
Converted into code to
support the process (requires people)
Keynote at ICSE’08: Herbert Hanselmann: Challenges in
Automotive Software Engineering
“Model-based design (MBD) of functional behaviour has been a
big help in the recent past”
MasterMind
Neches, et.al., IUI’93
One of the first system to integrate multiple models
together
Mobi-D
Angel Puerta, IUI’97
Set of tools to support
a clearly defined
development cycle
Uses a series of
different models
Explicit relationships
that specify how the
models are related to
each other
Explicit interaction
between end users
and developers
Mobi-D Models
User-task Model
Domain Model
Describes the human-computer conversation at a low level
Presentation Model
Models of the entities that are manipulated in an interface and their
properties
Dialog Model
Describes tasks the user performs and what interaction capabilities are
needed
Specifies how the interaction objects appear in all of the different states
of the interface.
Relations
Describes how each of the models relate to each other
Tasks/Domain, Dialog/Presentation, Tasks/Dialog, etc.
Mobi-D Process
1. Elicit user tasks
Start with informal description and then convert to outline (basis for task models in
Mobi-D)
More formal task analysis methods could probably be used
2. User-task and domain modeling
Skeleton domain model is built from task outline
Developer refines model
Explicit methods for generalizing pieces of model for reuse in other interface designs
3. Presentation and Dialog design
Decision support tools provide recommendations from model and interface guidelines
(if available)
System steps through task model and helps designer build final interface
By carrying the task model through the process, Mobi-D’s designers believe
users can provide more useful feedback
Mobi-D Discussion
Provides assistance rather than automating design
Recommendations do not limit flexibility, but
organize the design process
For usability engineers, not everyday users
Benefits come from reuse among small projects or for
managing interaction data from a large project
Models can be large and appear to require significant effort
to develop
Spawned a profitable company http://www.redwhale.com/
that does UI work
XIML
eXtensible Interface Markup Language
XIML.org
Based on Mobi-D work
Supports full development lifecycle
Used by RedWhale Software to drive
their interface consultant business
They have developed many tools
move interaction data to/from XIML
Leverage data in XIML to better
understand various interfaces
Automate parts of the interface design
process
Example Use for XIML
Multi-platform interface development
Other Systems
UIML
Originally a research project at Virginia Tech, now being developed commercially
by Harmonia
Goal is platform independent language for describing UI
Early versions were not very platform independent
Recent versions using task models to automatically generate parts of the old
language that were not platform independent
Teresa
(http://www.harmonia.com/)
(http://giove.cnuce.cnr.it/teresa.html)
Transformation Environment for inteRacti
Tool for taking ConcurTaskTrees models, building an abstract interface, and then
building a concrete interface on multiple platforms.
USIXML
(http://www.usixml.org)
Many of the same features of XIML
Novel aspect is the use of graph structure for modeling relations (seems very
complex)
Ubiquitous Computing
Approach
“Pervasive computing cannot succeed if every device
must be accompanied by its own interactive software
and hardware…What is needed is a universal
interactive service protocol to which any compliant
interactive client can connect and access any service.”
-Dan Olsen (Xweb paper)
The web comes close to solving this problem, but is
interactively insufficient.
Ubiquitous Computing
Approach
There are two problems here:
Infrastructure issues
How do devices communicate?
How do devices discover each other?
User Interfaces issues
Are devices described sufficiently to build a good UI?
How are interfaces generated?
How can one interface be created for controlling
combinations of related devices?
Infrastructure Issues
Possible to investigate these issues without
automatically generating UIs
Being addressed by lots of systems
Commercially
Research
UPnP, JINI, Salutation, HAVi
Speakeasy (PARC), many others…
Most systems that address the UI issues also have
some infrastructure component
Systems addressing UI issues
XWeb
ICrafter
A system for integrating user interfaces from multiple
devices
Supple
Now known as ICE – Interactive Computing Everywhere
A system for automatically generating interfaces with a
focus on customization/personalization.
Personal Universal Controller
Jeff Nichol’s research…
XWeb
Work by Dan Olsen and group at BYU
E.g. UIST’2000, pp.191 - 200
Premise:
Apply the web metaphor to services in general
Support higher levels of interactivity
XWeb Protocols
Based upon the architecture of the web
XTP Interaction Protocol
Server-side data has a tree structure
Structured Data in XML
URLs for location of objects
xweb://my.site/games/chess/3/@winner
xweb://automate.home/lights/livingroom/
xweb://automate.home/lights/familyroom/-1
XWeb & XTP
CHANGE message (similar to GET in HTTP)
Sequence of editing operations to apply to a sub-tree
Set an attribute’s value
Delete an attribute
Change some child object to a new value
Insert a new child object
Move a subtree to a new location
Copy a subtree to a new location
Platform Independent
Interfaces
Two models are specified
DataView – The attributes of the service
XView – A mapping of the attributes into high-level “interactors”
Atomic
Numeric
Time
Date
Enumeration
Text
Links
Aggregation
Group
List
XWeb Example
DataView
Xweb
Example
XView
XWeb Example
Interface
Other XWeb Details
Has simple approach for
adjusting to different screen
sizes
Shrink portions of the
interface
Add additional columns of
widgets
Also capable of generating
speech interfaces
Based on a tree traversal
approach like Universal
Speech Interfaces
ICrafter
Part of the Interactive Workspaces research project
at Stanford
Ponnekanti, et. al. Ubicomp’2001
Main objective:
“to allow users of interactive workspaces to flexibly interact
with services”
Contribution
An intelligent infrastructure to find services, aggregate
them into a single interface, and generate an interface for
the aggregate service.
In practice, much of the interface generation is done by
hand though automatic generation is supported.
ICrafter Architecture
How is aggregation
accomplished?
High-level service interfaces (programmatic)
Data Producer
Data Consumer
The Interface Manager has pattern
generators
Recognize patterns in the services used
Generate interfaces for these patterns
This means that unique functionality will not be
available in the aggregate interface
Automatic Generation in
ICrafter
Manual Generation in ICrafter
Supple
Eventual goal is to support automatic
personalization of user interfaces
Treats generation of interfaces as an
optimization problem
Can take into account usage patterns in
generation
Krzysztof Gajos and Daniel S. Weld, “SUPPLE: Automatically Generating User Interfaces” in
Proceedings of Intelligent User Interfaces 2004, Funchal, Portugal.
Modeling Users with Traces
Supple uses traces to keep a usage model
Sequences of events:
<interface element, old value, new value>
Interfaces are rendered taking the traces into
account (though traces are not required)
Trails are segmented at interface close or reset
Generating with Optimization
Uses a branch-andbound search to
explore space of
alternatives
Guaranteed to find
an optimal solution
Optimization Equation
Cost of navigation between
subsequent controls in a trace
Device-specific measure
of how appropriate a
control is for manipulating
a variable of a given type
Total cost for one user trace
Total cost of all traces
Screenshots
Personal Universal Controller
Jeff Nichol’s PhD work
Problem:
Appliance interfaces are too complex and too
idiosyncratic.
Solution:
Separate the interface from the appliance and use
a device with a richer interface to control the
appliance:
PDA, mobile phone, etc.
Approach
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
The PUC System Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
Automatic Generation of UIs
Benefits
All interfaces consistent for a user
With conventions of the handheld
Even from multiple manufacturers
Addresses hotel alarm clock problem
Can take into account user preferences
Multiple modalities (GUI + Speech UI)
A Hard Problem
Previous automatic systems have not
generated high quality interfaces
PUC Specification Language
XML
Full documentation for the
specification language and
protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample
specification for a stereo
Properties of PUC Language
State variables & commands
Each can have multiple labels
Typed variables
Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure
Groups
Dependency Information
Crucial for high-quality interfaces
Expressed as <active-if> clauses
Operations:
Combined Logically
Equals, Less-Than,
Greater-Than
AND, OR
Used for:
Dynamic graying out
Layout
Widget selection
Specifications
Have working specifications for:
Audiophase stereo
X-10 lights control
Sony CamCorder
Windows Media Player
Audio ReQuest hardware MP3 player
WinAmp Media Player
Elevator
Parts of GMC Yukon Denali SUV
Etc.
Controller Generators
iPaq PocketPC
SmartPhone
No touchscreen
Desktop (TabletPC)
Speech
Generating Speech Interfaces
“Universal Speech Interface” (USI) project
Creates grammar, language model and
pronunciation dictionary from PUC specification
Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Will use dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback
Paraphrases as confirmation
Adaptors
“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
AV/C (standard protocol)
Sony CamCorder
Custom software
Lutron
HAVi
Windows Media Player
UPnP
X-10
Axis Camera
Light switches, etc.
Generating Consistent UIs
Personally consistent
Reduce learning time
Add unique functions
Generating Combined UIs
For multiple appliances, such as home
theaters
Specify content flow
Combined controls
Summative Study
Compared PUC to manufacturer’s
interfaces for HP and Canon
printer/fax/copiers
PUC twice as fast, 1/3 the errors
Consistent: another factor of 2 faster
18
16
1200
Number of Failures
Average Time (sec)
1400
1000
800
600
400
14
12
10
8
6
4
200
2
0
0
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
CanonConsistent
HP
HP-PUC
HPConsistent
Canon
Canon-PUC
CanonConsistent
Details of the Language Design
Informed by hand-designed interfaces
What functional information was
needed to create interfaces?
Additional Requirements
Support complete functionality of
appliance
No specific layout information
Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
Language Elements
Elements
State variables & commands
Labels
Group tree
Dependency information
Example media player
specification
Play, stop, pause, next track,
previous track
Play list
Language Elements,
cont.
State Variables and
Commands
Represent functions of
appliance
State variables have types
Boolean, Enumeration,
Integer, String, etc.
Variables sufficient for most
functions but not all
e.g. “seek” button on a Radio
Language Elements, cont.
Label Information
One label not suitable everywhere
The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”
A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements,
cont.
Label Information
One label not suitable everywhere
The optimal label length
changes with screen size
Speech interfaces may benefit
from pronunciation and
text-to-speech information
“Label Dictionary”
A group of semantically similar
labels
Different lengths
Information for different
modalities
Language Elements, cont.
Group Tree
Specify organization
of functions
We use n-ary tree
with variables or
commands at leaves
Also used for
specifying complex
types
Lists
Unions
Language Elements,
cont.
Group Tree
Specify organization
of functions
We use n-ary tree
with variables or
commands at leaves
Also used for
specifying complex
types
Lists
Unions
Language Elements,
cont.
Dependency Information
Formulas that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less
Than, Is Defined
Linked with logical operators
(AND, OR)
Allows feedback to user when
a function is not available
Graphical Interface Generator
Rule-based approach
Multiple phases that
iteratively transform a
specification into a
user interface
Focuses on panel structure of
user interface
Small groups of controls have
basic layouts
Complexity comes from
structure of groups
Structure can be inferred
from dependency info!
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
Generation Process
1. Determine conceptual layout
Infer panel structure from
dependencies using “mutual
exclusion” property
Choose controls (decision tree)
Choose row layout
Without layout fixing rules
(one column, two column, etc.)
2. Allocate space
Examine panel contents and
choose sizes
3. Instantiate and place controls
4. Fix layout problems
With layout fixing rules