Transcript Automatically Generating High-Quality User Interfaces for Appliances Jeffrey Nichols Doctoral Symposium Presentation Seventeenth Annual ACM Symposium on User Interface Software and Technology October 24, 2004

Automatically Generating High-Quality
User Interfaces for Appliances
Jeffrey Nichols
Doctoral Symposium Presentation
Seventeenth Annual ACM Symposium on User Interface Software and Technology
October 24, 2004
Problem
Appliances are complex and their user interfaces are often hard to use!
1
Approach: Use Mobile Devices
Built for general-purpose interaction
Common, cheap, and often have ability to communicate
10 million smart phones sold
11.2 million handheld units shipped in 2003
120 million mobile phone subscribers in US
2
Approach, cont.
Use mobile devices to control all appliances in the environment
Specifications
Control
Feedback
Appliances
Mobile Devices
Key Features
Two-way communication, Abstract Descriptions, Multiple Platforms,
Automatic Interface Generation
3
Why Automatic Generation?
Benefits
•
•
Multiple platforms and modalities (GUI + Speech UI)
All interfaces consistent for a user
•
•
•
With conventions of the handheld
Even from multiple manufacturers
Create user interfaces that control multiple connected
appliances
e.g. a home theater
4
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
5
Research Approach
1. Hand-design remote control interfaces
2. Determine functional information needed
from appliances to design user interfaces
3. Design language for describing appliance
functions
4. Build interface generators for multiple
platforms
5. Perform user studies to evaluate the
interface generators
6
Initial Approach
What information is needed about the
appliance to automatically generate
remote control interfaces?
Investigate via a design process
Create interfaces by hand
AIWA Shelf Stereo
• AT&T Telephone/Answering Machine
•
Improve quality with heuristic analysis and
think-aloud studies with several users
Compare interfaces with actual appliance
interfaces to validate PUC concept
Analyze interfaces for functional information
7
Language Design
Informed by hand-designed interfaces
•
What functional information was needed
to create interfaces?
Additional Requirements
•
Support complete functionality of
appliance
•
No specific layout information
•
Only one way to specify anything
Full documentation available at: http://www.cs.cmu.edu/~pebbles/puc/
8
Language Elements
Elements
•
•
•
•
State variables & commands
Labels
Group tree
Dependency information
Example media player specification
•
•
Play, stop, pause, next track,
previous track
Play list
9
Language Elements, cont.
State Variables and Commands
•
Represent functions of
appliance
•
State variables have types
•
•
Boolean, Enumeration, Integer,
String, etc.
Variables sufficient for most
functions but not all
•
e.g. “seek” button on a Radio
10
Language Elements, cont.
Label Information
One label not suitable everywhere
•
•
The optimal label length changes
with screen size
Speech interfaces may benefit from
pronunciation and
text-to-speech information
“Label Dictionary”
•
•
•
A group of semantically similar labels
Different lengths
Information for different modalities
11
Language Elements, cont.
Label Information
One label not suitable everywhere
•
•
The optimal label length changes
with screen size
Speech interfaces may benefit from
pronunciation and
text-to-speech information
“Label Dictionary”
•
•
•
A group of semantically similar labels
Different lengths
Information for different modalities
12
Language Elements, cont.
Group Tree
•
Specify organization of
functions
•
We use n-ary tree with
variables or commands
at leaves
•
Also used for specifying
complex types
Lists
Unions
13
Language Elements, cont.
Group Tree
•
Specify organization of
functions
•
We use n-ary tree with
variables or commands
at leaves
•
Also used for specifying
complex types
Lists
Unions
14
Language Elements, cont.
Dependency Information
• Formulas
that specify when a
variable or command is active
in terms of other state
variables
Equals, Greater Than, Less Than,
Is Defined
Linked with logical operators
(AND, OR)
• Allows
feedback to user when a
function is not available
15
Interface Generators
Generators for Two Modalities
Graphical
•
Desktop, PocketPC, and
Microsoft Smartphone
Speech
•
•
•
Collaboration with others at
Carnegie Mellon
Built on top of the PUC
framework
Implemented using Universal
Speech Interface (USI)
techniques [Rosenfeld 2001]
16
The PUC System Architecture
APPLIANCES
(Stereo, Alarm Clock, etc.)
PUC DEVICES
(automatic interface generation)
ADAPTOR
(publishes description +
appliance state + controls appliance)
PROTOCOL
(two-way communication
of specification & state)
device specification &
state feedback
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
PROTOCOL
(two-way communication
of specification & state)
COMMUNICATION
(802.11, Bluetooth, Zigbee, etc.)
control
17
Controlling Appliances
We have built adaptors for many actual appliances
•
•
•
•
•
•
•
•
•
Sony Digital Camcorder
Windows Media Player
Axis UPnP Pan & Tilt Camera
Lutron Lighting
X10 Lighting
Audiophase Shelf Stereo
AudioReQuest MP3 player
GM Vehicle Information System
GM Vehicle Climate Control
Written specifications for others
•
•
•
•
Elevator
Telephone/Answering Machine
GM Navigation System
Several Alarm Clocks
18
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
19
High-Level Conventions
Problem
•
Human designers
rely partly on
conventions when
making an
interface
•
Users expect their
appliances to use
conventions they
know about
•
How do we
integrate highlevel info into
PUC system?
20
Smart Templates
Method for specifying high-level
information to interface generators
Solution
•
Mark groups with tags that identify highlevel information
media-controls, phone-dialpad, time,
date, etc.
•
Restrict the contents of groups so that
interface generators can interpret the
high-level meaning
•
Standardize the tags and restrictions in
advance, so that designers know what
interface generators expect
21
Smart Templates, cont.
Features
• Parameterized
• Specified
using primitive elements
of specification language
Renderable by any interface generator
• Generators
can contain special
code that renders a template as an
interface designer might
22
Interfaces with Smart Templates
Preliminary Implementation
A few templates: image, image-list,
media-controls, time-duration
23
Continuing Work
Define and implement Smart Templates
•
•
•
date, mute, power, time-absolute, volume, etc.
Use to improve renderings of list interfaces
Develop more as more appliances are specified
Combinations of templates
•
•
Less implementation cost than a new template
e.g. date and time-absolute
Use of templates with data already on controller device
•
•
e.g. calendar and address information
Might allow user to enter address from contact list into
navigation system
24
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
25
Interface Consistency
PUC devices have a unique opportunity to provide
consistency for the user
•
•
Personal device
Used for interacting with most appliances
Two ways that PUC UIs can be made consistent
•
•
With other applications on the same device
With past interfaces for similar appliances
26
Consistency with Past Interfaces
Two sub-problems to address:
•
Similarity
Which functions of a new appliance are similar to the
functions of interfaces generated in the past?
•
Consistency
For similar functions, what rules from previous interfaces
can be applied to ensure consistency?
27
Similarity Problem
Difficult to conclusively know whether two functions from different
appliances are the same
•
Very little semantic information in the specification language
Can estimate similarity based on properties of state variables
•
•
•
•
•
Smart Template
Name
Group Name
Labels
Type
Improve estimate by looking at relationships between multiple
similar variables
28
Similarity & Consistency Problems
new
previous
sparse similarity
new
previous
branch similarity
new
previous
significant similarity
29
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
30
The “Experience”
31
The “Experience”, cont. (MIT dorm)
How can a PUC device provide improved
interfaces for connected systems?
32
Improving Interfaces with PUC
Generate an interface that aggregates all functions into
one set of screens
Organized by task instead of appliance
Automatically create macros for frequently used functions
e.g. “Play DVD” would:
1.
2.
3.
4.
5.
turn on television, DVD player, stereo
turn off VCR
set the TV and stereo sources to the DVD player
instruct the user to insert a DVD (if necessary)
play the DVD
How can appliance descriptions support these features?
33
Model Data Flow
Define inputs and outputs of each appliance
Including definition of whether output is human
consumable (e.g. screen, speakers, etc.)
• May need to define which appliances are content
sources and sinks
•
Develop a basic language for describing how
states and commands modify I/O
Accept/Reject input i
• Activate/Modify/Deactivate output o
•
34
Distributed Task Language
Data flow helps us find cross-appliance functions,
but does not tell us much about likelihood of use
•
Need to be able to filter
Add some task information
•
Store task information within each appliance
description
Use an existing task language
•
Combine sub-tasks from each appliance in a
connected system to create complete tasks
35
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
36
Two Goals for PUC System
Breadth
The appliance specification language is capable of
describing a wide variety of appliances
Quality
Interfaces generated for specifications across that
range beat the usability of the manufacturers’
interfaces for the same appliances
How do I validate that these goals are met?
37
Evaluation
Evaluation Method
1.
Develop a list of appliances that are interesting
Complexity
• Unique feature
• Representative of a class of appliances
•
Specify each appliance
3. Perform a comparative user study on several
appliances to test quality
2.
38
Evaluation, cont.
Interface consistency
•
Train users to be experts with one interface and test
performance on a related appliance interface
Control for consistency features
•
Test the variability in appliance specifications
Multi-appliance UIs
•
Compare performance of users with/without multiappliance features
Focus on structure, selection of multi-appliance functions
39
Outline
•
Introduction
•
Personal Universal Controller (PUC) System
•
Handling High-Level Conventions
•
Interface Consistency
•
Generating Interfaces for the “Experience”
•
Validation
•
Conclusion
40
Conclusion
Problem
•
Appliances are increasingly complex
•
Appliance user interfaces are often hard to use
Solution
•
Move appliance interfaces to a mobile device the user is
already carrying
•
Automatically generate interfaces so that:
Interfaces are customized for the device and modality that the user prefers
Interfaces for similar appliances can be made consistent
Interfaces for multiple appliances can be combined into a single interface
•
Validate generated interfaces to prove high quality claim
41
Acknowledgements
Thesis Committee
•
•
•
•
Brad A. Myers (chair)
Scott Hudson
John Zimmerman
Dan Olsen Jr.
Funding
•
•
•
•
•
National Science Foundation
Microsoft
General Motors
Intel
Pittsburgh Digital Greenhouse
Equipment Grants
•
•
•
•
•
•
Mitsubishi (MERL)
VividLogic
Lucent
Lutron
Lantronix
Nokia
PUC Project Members
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Kevin Litwack
Thomas K. Harris
Michael Higgins
Joseph Hughes
Roni Rosenfeld
Rajesh Seenichamy
Pegeen Shen
Htet Htet Aung
Mathilde Pignol
Suporn Pongnumkul
Stefanie Shriver
Jeffrey Stylos
Peter Lucas
Thomas Psik
Collaborators & Friends
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Naomi Ramos
Desney Tan
Daniel Avrahami
Gaetano Borriello
Laura Dabbish
Andrew Faulring
James Fogarty
Krzysztof Gajos
Darren Gergle
Andy Ko
Amy Nichols
Mick Nichols
Sally Nichols
Trevor Pering
Fleming Seay
Irina Shklovski
Roy Want
Jake Wobbrock
and many others…
42
Thanks for listening!
For more information…
http://www.cs.cmu.edu/~pebbles/puc/
http://www.cs.cmu.edu/~jeffreyn/
Automatically Generating High-Quality
User Interfaces for Appliances
Jeffrey Nichols
Doctoral Symposium Presentation
Seventeenth Annual ACM Symposium on User Interface Software and Technology
October 24, 2004
45
46
47
XML-based Specification Language
Describes appliance with these features:
•
Functions of Device
State Variables and Commands
•
Labeling
Multiple labels are necessary
•
Grouping
Hierarchical groups
•
Dependency Information
For enabling and structure
48
Outside the Scope
•
•
•
•
•
•
Help systems for generated interfaces
Automated trouble-shooting for complex systems
Service Discovery
Macros and End-User Programming
Security
Inter-operability with INCITS/V2
49
Hand-Designed Interfaces
Interfaces for Microsoft’s PocketPC (simulated remote control)
telephone
stereo
Using our interfaces, users were twice as fast and made half as many errors
50
Comparison Study
Compared performance of first-time users (not experts)
Appliance
Hand-design
Phone
Stereo
Procedure
•
Each subject worked two sets of tasks
both stereo and phone
controlled for order and interface
Performance Metrics
•
Time to complete all tasks
•
Number of times a user manual was needed
•
Number of missteps
51
Comparison Study Results
Using our interfaces, users were twice as fast
and made half as many errors
All differences are significant (p < 0.05)
52