Transcript Generating Remote Control Interfaces for Complex Appliances

Generating
Remote Control Interfaces
for Complex Appliances
Jeffrey Nichols
Carnegie Mellon University
August 22, 2002
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #0
The Problem
Everyday appliances are becoming more
complicated and thus more difficult to use
 Cheap to add computation and features
 Expensive to design and build high-quality interfaces
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #1
The Solution
Observation:
People are increasingly carrying handheld devices,
like mobile phones and PDAs
 These devices will soon be able to communicate with
devices around them (Bluetooth, 802.11b)
 Better built-in UI hardware (color touch screens, etc.)
Solution:
Use handhelds as remote controls!
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #2
Personal Universal Controller
An intermediary remote control interface
Specifications
Control
Feedback
Benefits
 Graphical and speech interfaces
 Adaptable interfaces (to PUC device and to user)
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #3
Overview
Architecture
Specification Language
Interface Generators
 Algorithms using dependency information
Demonstration
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #4
Architecture
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #5
Appliance Adaptors
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #6
Communication Protocol
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #7
Interface Generators
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #8
Specification Language
XML-based!!!
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #9
Language Design
Approach
 Create reference interfaces
 AIWA Shelf Stereo
 AT&T Telephone/Answering Machine
 Test interfaces with subjects
 Users twice as fast and made half
errors as compared with
manufacturer’s interfaces
 Analyze interfaces for functional
information
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #10
Language Elements
State Variables and Commands
 Represent functions of stereo
 Variables sufficient for most functions but not all
 “seek” button on a Radio
Type Information
 State variables have types
 Boolean, Enumeration, Integer, String, etc.
 Interface generators can use type to infer control
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #11
Language Elements, cont.
Label Information
 Good labels are an important of any interface
 One label does not fit all
 The optimal label size changes with screen size
 Speech interfaces made need pronunciation and textto-speech information
 Solution: Label Dictionary
 Collection of similar labels
 As many as necessary
 Different modality information
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #12
Language Elements, cont.
Group Tree
 Organization is very
important to user
interfaces
 Tree is a good
structure as most
interfaces can be
described as a tree
 We use n-ary tree with
variables/commands at
leaves
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #13
Language Elements, cont.
Dependency Information
 Formulas that specify when a variable or
command is active in terms of other state
variables
Equals, Greater Than, Less Than
Linked with logical operators (AND, OR)
 For example,
<and>
<equals state=“PowerState”>true</equals>
<equals state=“RadioBand”>AM</equals>
</and>
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #14
Overview
Architecture
Specification Language
Interface Generators
 Algorithms using dependency information
Demonstration
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #15
Graphical Interface Generator
Focuses on panel structure of
user interface
 Small groups of controls have
simple layouts
 Complexity comes from relative
layouts of groups
Structure can be inferred from dependency info
 Groups are often defined by sets of controls that
are all active at the same time (mutual exclusion)
 May overlap other controls that not active
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #16
Determining Panel Structure
1. Find mutual exclusion within group tree


Find group that contains
depended-upon state
Look for mutual exclusions
among group children
2. Assign branches of group
tree to sets of panels

Choose a panel based upon
type information and properties
of group tree.
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #17
Choosing Panel Types
Rules for Choosing Panels
 If state has enumerated type and
there are mutually exclusive groups
of controls for each value, create a
tabbed panel.
 If state has boolean type and all other
controls are active for only one value,
create two full-screen overlapping
panels.
 Create overlapping panels for each
mutually exclusive group and ensure
there is a control elsewhere for
setting the state.
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #18
Making the Interface Concrete
Choose a control for each object
Use a decision tree with questions
such as:
 Is this a command or state variable?
 What is the type of the state
variable?
 Is the state variable read-only?
 Was a panel structure rule applied
from this variable?
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #19
Making Concrete, cont.
Rules for choosing row layouts
within panels
 A labeled group with only two
unlabeled controls is given a twocolumn layout with group label
 Controls that prefer a lot of
horizontal room are given a full
width layout
 If a group contains just two labeled
controls are given a two-column
layout without group label
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #20
Making Concrete, cont.
Conceptual layout has been chosen
 Panels, Controls, Rows
Allocate space
 Examine panel contents and choose sizes
Instantiate and place controls
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #21
Overview
Architecture
Specification Language
Interface Generators
 Algorithms using dependency information
Demonstration
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #22
Future Work
 Improve quality of generated user interfaces
 Back-tracking, etc.
 Specification and generation of list interfaces
 Necessary for many appliances
 Adaptive interface generation
 New interfaces modeled on past interfaces
 General purpose appliance adaptor for HAVi
and UPnP protocols
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #23
Acknowledgements
PUC Project Members
 Brad A. Myers
 Michael Higgins
 Joseph Hughes
 Thomas K. Harris
 Roni Rosenfeld
 Stefanie Shriver
 Peter Lucas
 Mathilde Pignol
 Kevin Litwack
Funding
 National Science Foundation
 Pittsburgh Digital Greenhouse
 Microsoft
Equipment Grants





Mitsubishi (MERL)
VividLogic
Symbol Technologies
Hewlett-Packard
Lucent
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #24
Thanks!
http://www.cs.cmu.edu/~pebbles/puc/
http://www.cs.cmu.edu/~jeffreyn/
Generating Remote Control Interfaces for Complex Appliances
August 22, 2002
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #25
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #26
Framework for Adaptors
PUCProxy
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #27
Control of IEEE 1394 DV Camera
Sony Camcorder with AV/C protocol
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #28
Control of Audiophase Stereo
Custom Hardware and Software Adaptor
Continue to Future Work
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #29
Interface Generation Process
Three Phases
 Determine panel structure from dependency info
 Assign controls to variables/commands
 Allocate space and lay out controls
Jeffrey Nichols • Generating Remote Controls for Complex Appliances • August 22, 2002 • Slide #30