Human Computer Interaction Institute School of Computer Science Carnegie Mellon University An Update on the Pebbles Project: PocketPCs, Smartphones and TabletPCs for Universal Remote Control and A.
Download
Report
Transcript Human Computer Interaction Institute School of Computer Science Carnegie Mellon University An Update on the Pebbles Project: PocketPCs, Smartphones and TabletPCs for Universal Remote Control and A.
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
An Update on the Pebbles Project:
PocketPCs, Smartphones and
TabletPCs for Universal Remote Control
and
A New Input Method for Small Devices
Brad A. Myers
[email protected]
http://www.cs.cmu.edu/~pebbles
July 30, 2003
Pebbles Project
Use of multiple devices at the same time
Pocket PC and desktop PC
Mobile Phone and “Smart Home”
Multiple handhelds in a meeting
Pocket PC and appliances
Multiple users with their devices
Single user with multiple devices
Brad Myers
2
Handhelds are communicating
802.11
BlueTooth
Cell-phone network (G3)
(Infrared)
(Wires or cradle)
Brad Myers
3
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
4
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.cmu.edu/
Brad Myers
5
Previously Reported Here
Microsoft has supported this research since ‘98
Prior presentations here about Pebbles:
For business meetings:
To augment desktop applications:
For individuals
For classrooms
For group work
How laser pointers can be integrated
To make lectures more interactive
For military command posts:
To facilitate communication and collaboration
Brad Myers
6
Example: Slide Show Commander
Released 1999
Remote Control of PowerPoint
Two-way communication
Picture of slide, notes, titles,
other applications, preview
Scribble, click on links
Brad Myers
7
Slide Show Commander, #2
List of applications
Preview of other slides
(Soon: continuous preview ahead)
Brad Myers
8
Download SlideShow Commander
Formerly was for sale, but no longer
Research version distribution limited to
academics and Pebbles project supporters,
including Microsoft employees:
http://www.pebbles.hcii.cmu.edu/private/sscpocketpc
Brad Myers
9
Example 2: Shortcutter
User-created panels of controls
Create custom interfaces and
extensions to PC applications
Augment desktop applications
Useful for prototyping
Example: two handed interaction:
scrolling with left hand
Direct manipulation for edit, then set
properties
Brad Myers
10
Shortcutter Widgets
Buttons
Virtual Joystick
Virtual Knob
Sliders
Mouse pad
Gesture panel
Brad Myers
11
Scenarios of Use
Lean-back mail reading
Controlling media players
… and many others
Brad Myers
12
Personal Universal Controller
PhD research of Jeffrey Nichols
http://www.pebbles.hcii.cmu.edu/puc/
Today’s Talk
Personal Universal Controller
Last year, reported first steps
New this year:
Tablet PC, Smartphone
New devices: UPnP, WMP, Lutron, etc.
Ported to C# from Java
Portability across devices
Better quality UIs due to “high-level templates”
Brad Myers
14
Problem
Appliances are too complex
Brad Myers
15
Problem
Too many remotes
Brad Myers
16
Problem
April 8, 1991
Brad Myers
17
Approach
Use a personal handheld device as an
interface to the appliances
Brad Myers
18
Why Appliances Are Hard to Use
Too many complex devices, each with its
own idiosyncratic interface
Stereo system
Telephones
ATM
Fax machine
Photocopier
Hotel alarm clock
Increasingly computerized
Brad Myers
19
Existing “Universal” Controls
Pre-programmed at the factory with a subset,
Or, Laboriously hand-programmed by the user
Brad Myers
20
Our Approach
Two-way communication
Appliance describes its functions
Handheld PUC
Automatically creates interface
Controls the appliance
Displays feedback about appliance status
Specifications
Control
Feedback
Brad Myers
21
Automatic Generation of UIs
Benefits
All interfaces consistent on the controller
With conventions of the handheld
Even from multiple manufacturers
Addresses hotel alarm clock problem
Can take into account user preferences
Multiple modalities (GUI + Speech UI)
A Hard Problem
Previous automatic systems have not
generated high quality interfaces
Brad Myers
22
Initial User Study
Compared hand-designed interfaces to
manufacturers’
Stereo and telephone
Using PUC, users took 50% less time &
made 50% fewer errors
All differences are significant (p < 0.05)
Brad Myers
23
Properties of PUC Language
XML
State variables & commands
Each can have multiple labels
Typed variables
Useful when not enough room
Base types: Boolean, string,
enumerated, integers,
fixed-point, floating-point, etc.
Optional labels for values
Hierarchical Structure
Groups
Brad Myers
24
Dependency Information
Crucial for high-quality interfaces
Expressed as <active-if> clauses
Operations:
Combined Logically
Equals, Less-Than,
Greater-Than
AND, OR
Used for:
Layout
Widget selection
Dynamic graying out
Brad Myers
25
Specifications
Have working specifications for:
Windows Media Player
Simulated Elevator
Axis Pan-Tilt-Zoom camera through UPnP
GMC Denali Driver Information Console
GMC Denali Climate Control System
Key:
GMC Denali Navigation System
Demonstrate
Lutron lighting system
today
Sony CamCorder
X-10 lights control
Demonstrated
Audiophase stereo
at Faculty
Audio ReQuest hardware MP3 player
Summit
WinAmp Media Player
Demonstrated
UPnP Light Control
here last year26
Brad Myers
Generating Speech Interfaces
“Universal Speech Interface” (USI) project
Creates grammar, language model and
pronunciation dictionary from PUC specification
Prof. Roni Rosenfeld of CMU
http://www.cs.cmu.edu/~usi
Pronunciation from labels using phonetic rules
Can provide other pronunciations as labels for fine-tuning
Uses dependency information to help with
disambiguation and explanation
Supports queries and spoken feedback
Paraphrases as confirmation
Brad Myers
27
New Work: C#
Ported to C# with .NET Compact Framework
With support from Microsoft – Thanks!
Much faster execution than Java version on
PocketPC
Got portability across PocketPC,
SmartPhone, TabletPC (desktop)
Significant code reuse
90% from PocketPC to TabletPC
60% from PocketPC to SmartPhone
Different input model on SmartPhone requires
different generator rules and different widgets
Brad Myers
28
New Work: High-Level Templates
Templates for “high-level” types
Play control
Time/date
Light on, dim-level, off
Issue: Highly parameterized
E.g., what options available for play group
Tape: play, pause, stop, FF, Rew
CD: play, pause, stop, FF, Rew, next track, prev. track
Live audio feed: play, stop
Answering machine: play, play new, stop, …
Brad Myers
29
High-Level Templates
If PUC generator doesn’t understand
template, then falls back to primitive types
Primitive types also provide parameterization
CD:
Camera:
Brad Myers
30
Adaptors
“Adaptors” provide the interface to existing
(and future) appliances
If do not support specification language directly
Custom hardware
AV/C (standard protocol)
Sony CamCorder
Custom software
Lutron
HAVi
Windows Media Player
UPnP
X-10
Axis Camera
Light switches, etc.
Brad Myers
31
New Work: New Devices
UPnP interface to Axis Camera
Lutron light interface
General Motors automobile
Windows Media Player
Simulated elevator, etc.
Brad Myers
32
Demonstration
New devices:
Windows Media Player through COM
Axis Camera through UPnP
Simulated Elevator
Simulated GMC Denali systems
Non-driving functions
GUI and speech
Cross platform: PocketPC, SmartPhone, TabletPC
Two way communication:
When state changes from appliance, GUI is updated and
can query state with speech
Brad Myers
33
Example Screens
Axis Camera
Brad Myers
Windows Media Player
34
Future Work
Remote Control for “Experiences”
Combined control of multiple appliances
Better quality generation
New devices?
Automatic creation of specifications for
UPnP devices?
Brad Myers
35
EdgeWrite™
PhD research of Jake Wobbrock
(patent pending)
http://www.pebbles.hcii.cmu.edu/assistive/
Origins
Prior work on using handhelds as input
devices for people with Muscular Dystrophy
Didn’t generalize to other disabilities
Brad Myers
37
Idea: Use Edges for Stability
First design: use edges around entire screen
User study: moving along edges and into
corners much more accurate
Especially for people with disabilities
Brad Myers
38
Initial Design
Place letters around top and sides of screen
Alphabetic layout and by letter frequency
Not effective
Distances too long
Character sizes too small
Visual search times too large
Requires too much visual
attention
“the woman”
Brad Myers
39
Next Design: Smaller Area
Added custom template with a square hole
Designed alphabet using gestures inside of
hole, along edges and into corners
First implementation: for Palm
Brad Myers
40
Alphabet Properties
Recognizes based on sequence of corners
Users can wiggle during diagonals
Not speed dependent
Easy to customize and
adapt letter forms
Letters “feel” normal
Even user-driven
Use multiple strokes along edge
Presentation (only) uses arcs
No capitalization mode
Capitals end in upper left
Brad Myers
41
EdgeWrite Engineering
Change target area shapes/sizes
Start area vs. moving
Right-handed vs. left-handed
Plastic can’t sit on digitizer
Brad Myers
42
Current Alphabet Design
Also have
alternative
forms
104 unique
characters
288 total
character
forms
Brad Myers
43
Initial User Study
For able-bodied novices:
Just as learnable as Graffiti
Accuracy 18% better
Speed not significantly different
Brad Myers
44
User Study, Cont.
For handicapped users:
Succeeded with EdgeWrite when unable to use
Graffiti
Parkinson’s Disease
Cerebral Palsy
Muscular Dystrophy
For example:
“The g i gbsiangu% fast” vs:
“The dog is going fast”
Brad Myers
45
Future EdgeWrite Work
Test with Xbox joystick
Test with other devices (watch, phone, etc.)
Test long-term (expert) use
Test on unstable platforms (riding on a bus or car)
Etc….
1
2
4
5
7
*
Brad Myers
8
0
3
12:03
6
9
#
46
For More Information
Many papers on the Pebbles site
On PUC and other work
Shortcutter and Remote Commander
available for free downloading:
http://www.cs.cmu.edu/~pebbles
SlideShow Commander available to
Microsoft employees:
http://www.cs.cmu.edu/~pebbles/private/
Brad Myers
47
Thanks to Our Sponsors!
Supported by grants from:
DARPA
NSF
Microsoft
Pittsburgh Digital Greenhouse
General Motors
NEC Foundation of America
And equipment grants from:
Hewlett Packard
Lucent Technologies
Mitsubishi
Palm Computing
Symbol Technologies
IBM and IBM Canada
SMART Technologies
Brad Myers
VividLogic
Synergy Solutions
TDK Systems Europe, LTD
Lutron, Inc.
Lantronix, Inc.
Handango
48
Thanks to Pebbles Students
This research performed by a large number
of students since 1997:
Rishi Bhatnagar
Ben Bostwick
Franklin Chen
Yu Shan Chuang
Karen Cross
Carl Evankovich
Marc Khadpe
Dave Kong
Chun-Kwok Lee
Joonhwan Lee
Brad Myers
Mathilde Pignol
Suporn Pongnumkul
Jennifer Li
Leo Lie
Jack Lin
Kevin Litwack
A. Chris Long
Rob Miller
Jeff Nichols
Choon Hong Peck
Rajesh Seenichamy
Herbert Stiel
Jeff Stylos
Claire Tokar
Marsha Tjandra
Adrienne Warmack
Jacob O. Wobbrock
Jerry Yang
Sunny Yang
Brian Yeung
49
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
An Update on the Pebbles Project:
PocketPCs, Smartphones and
TabletPCs for Universal Remote Control
and
A New Input Method for Small Devices
Brad A. Myers
[email protected]
http://www.cs.cmu.edu/~pebbles
July 30, 2003
Black slide, rest are extra
Discussion of Comparison
Our hand designed interfaces
succeeded for several reasons
Good organization
Each button has one function
Good labels
Only available functions are active
Others, hidden on tabs or grayed out
Better feedback and error messages
Brad Myers
52
Current PUC Specification Language
XML
Full documentation for the
specification language
and protocol:
http://www.pebbles.hcii.cmu.edu/puc/
Contains sample
specification for
Audiophase stereo
Brad Myers
53
Examples of Generated GUIs
Stereo and X-10
Brad Myers
54