Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for Control Brad A.
Download ReportTranscript Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Handheld Devices for Control Brad A.
Human Computer Interaction Institute School of Computer Science Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
http://www.cs.cmu.edu/~pebbles
Handheld Devices
Handheld Devices for: Communication As a “Personal Information Manager” As a “Personal Digital Assistant” As a PC replacement “PocketPC” Handheld Devices for Remote Control other devices of Remote Control of PCs Remote Control of appliances Brad Myers Carnegie Mellon 2
Problem
Appliances are too complex Brad Myers Carnegie Mellon 3
Problem
Too many remotes Brad Myers Carnegie Mellon 4
Problem
April 8, 1991 Brad Myers Carnegie Mellon 5
Approach
Use a personal handheld device as an Interface to the PC and to the Appliances Brad Myers Carnegie Mellon 6
History of Communication
ParcTab – continuous communication Early “organizers” – no communication Newton – extra cost for synchronization Palm – HotSync once a day WindowsCE/PocketPC – ActiveSync once a day Brad Myers Carnegie Mellon 7
Now, Handhelds will be Communicating
Toshiba e740 802.11
BlueTooth Cell-phone network (G3) (Infrared) (Wires or cradle) HP iPaq 3870 Microsoft SmartPhone 8 Brad Myers Carnegie Mellon
Premises of our Research
“With the coming wireless technologies, connecting the PCs and PDAs together will no longer be an occasional event for synchronization. Instead, the devices will frequently be in
close, interactive communication
.”
—
Brad Myers, “Using Hand-Held Devices and PCs Together,” Comm. ACM, Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Brad Myers Carnegie Mellon 9
Pebbles Project
Performed as part of the
Pebbles
Project Overall goal: Use of multiple devices
at the same time
Palm
and
desktop PC Mobile Phone
and “
Smart Home” Multiple handhelds in a meeting Pocket PC
and
appliances Multiple users with their devices Single user with multiple devices Brad Myers Carnegie Mellon 10
Pebbles is:
P
DAs for
E
ntry of
B
oth
B
ytes and
L
ocations from
E
xternal
S
ources.
http://www.pebbles.hcii.edu/
Brad Myers Carnegie Mellon 11
Controlling a PC
Remote Commander Slide Show Commander “Semantic Snarfing” Shortcutter
http://www.pebbles.hcii.edu/
Remote Commander
Allow PDAs to control a PC Can be used with
any
application Uses the standard (single) cursor Don’t have to jump up and grab mouse All mouse and keyboard functions Use PDA like touchpad Graffiti or our own pop-up keyboard Word prediction and completion Brad Myers Carnegie Mellon 13
PocketPC version
Get PC’s screen onto PocketPC Full view, or one-to-one zooming Scroll with iPaq’s buttons Control or scribble Brad Myers Carnegie Mellon 14
For People with Muscular Disabilities Using handhelds as interface to PCs People with Muscular Dystrophy have fine-motor control but lose gross motor control Difficulties with mouse and keyboard, but stylus OK So use Remote Commander as PC’s keyboard and mouse Brad Myers Carnegie Mellon 15
NEW!
Alphabet for Accurate Gestures
Use a square hole in a template to increase accuracy Just as easy to learn as Graffiti ™ 18% more accurate for able novices (
p
<.02) Patent Pending Also works with joystick Brad Myers Carnegie Mellon 16
Shared Clipboard
Transfer information among your personal devices E.g., between PDA and PC Also between multiple computers Connect all clipboards together Transfer content or reference Works with
all
applications Brad Myers Carnegie Mellon 17
Power Point Control
Use PC to give the presentation Use hand-held to control the PC Two-way communication Hand-held shows picture of slide, notes, timer Brad Myers Carnegie Mellon 18
SlideShow Commander, cont.
See list of slide titles Scribble Tap on on-screen buttons and links Preview other slides Control demonstrations Brad Myers Carnegie Mellon 19
Control at a Distance
Controlling a PC from across the room Meetings, etc.
Use a laser pointer?
Studies “Semantic Snarfing” Grab contents to handheld Brad Myers Carnegie Mellon 20
Shortcutter
User-created panels of controls Create custom interfaces and extensions to PC applications And then take them with you Direct manipulation for edit, then set properties PocketPC or Palm Brad Myers Carnegie Mellon 21
Shortcutter Widgets
Buttons Virtual Joy stick Virtual Knob Sliders Mouse pad Gesture panel Brad Myers Carnegie Mellon 22
Scenarios of Use
Lean-back mail reading Controlling media players
… and many others
Brad Myers Carnegie Mellon 23
Controlling Appliances
PhD research of Jeffrey Nichols Assisted by undergrads and MS students
http://www.pebbles.hcii.edu/puc/
Problem
Too many complex devices, each with its own idiosyncratic interface Stereo system Telephones ATM Fax machine Photocopier Hotel alarm clock Increasingly computerized Low usability
Existing “universal” controls
Pre-programmed at the factory with a subset, Or, laboriously hand-programmed by the user Brad Myers Carnegie Mellon 26
Our Approach
Handheld is a “Personal Universal Controller” (PUC)
Two-way
communication Appliance describes its functions Handheld PUC
Automatically
creates interface Controls the appliance Displays feedback about appliance status Specifications Control Feedback Carnegie Mellon Brad Myers 27
Automatic Generation of UIs
Benefits All interfaces consistent for a user With conventions of the handheld Even from multiple manufacturers Addresses hotel alarm clock problem Can take into account user preferences Multiple modalities (GUI + Speech UI) A Hard Problem Previous automatic systems have not generated high quality interfaces Brad Myers Carnegie Mellon 28
Research Challenges
1.
Automatic
Design of Graphical User Interfaces
2.
Automatic
Design of Speech User Interfaces
3.
Connection
with real devices Through various protocols X-10, AV/C, HAVi, UPnP, etc. Also, custom hardware and software Brad Myers Carnegie Mellon 29
Hand-Generated Graphical Interfaces
First, Hand-Designed PocketPC interfaces: AIWA Shelf Stereo (Tape,CD,Tuner) AT&T Telephone/Answering Machine Used Embedded Visual Basic Ensured quality with heuristic analysis and think-aloud studies Compared with manufacturer’s interfaces Brad Myers Carnegie Mellon 30
Results of Comparison
Using PUC, users took 50% less time & made 50% fewer errors All differences are significant (
p
< 0.05) Brad Myers Carnegie Mellon 31
Current PUC Specification Language
XML Full documentation for the specification language and protocol: http://www.pebbles.hcii.cmu.edu/puc/ Contains sample specification for a stereo Brad Myers Carnegie Mellon 32
Properties of PUC Language
State variables & commands Each can have multiple labels Useful when not enough room Typed variables Base types: Boolean, string, enumerated, integers, fixed-point, floating-point, etc.
Optional labels for values Hierarchical Structure Groups Brad Myers Carnegie Mellon 33
Dependency Information
Crucial for high-quality interfaces Expressed as
Specifications
Have working specifications for: Audiophase stereo X-10 lights control Sony CamCorder Windows Media Player Audio ReQuest hardware MP3 player WinAmp Media Player Elevator Parts of GMC Yukon Denali SUV Etc.
Brad Myers Carnegie Mellon 35
Controller Generators
iPaq PocketPC SmartPhone No touchscreen Desktop (TabletPC) Speech Brad Myers Carnegie Mellon 36
Examples of Generated GUIs
Shelf stereo Windows Media Player Brad Myers Carnegie Mellon 37
Generating Speech Interfaces
“Universal Speech Interface” (USI) project Prof. Roni Rosenfeld of CMU http://www.cs.cmu.edu/~usi Creates grammar, language model and pronunciation dictionary from PUC specification Pronunciation from labels using phonetic rules Can provide other pronunciations as labels for fine-tuning Will use dependency information to help with disambiguation and explanation Supports queries and spoken feedback Paraphrases as confirmation Brad Myers Carnegie Mellon 38
Adaptors
“Adaptors” provide the interface to existing (and future) appliances Custom hardware If do not support specification language directly Custom software Lutron Windows Media Player X-10 Light switches, etc.
AV/C (standard protocol) Sony CamCorder HAVi UPnP Axis Camera Brad Myers Carnegie Mellon 39
Demonstration
Sony CamCorder Windows Media Player Simulated automobile Two way communication: When state changes from application or appliance, GUI is updated Can query state with speech Brad Myers Carnegie Mellon 40
Future Work on PUC
Controlling User “Experiences” One command for multiple devices Play a DVD sets DVD, cable, TV Setting alarm clock sets heat, coffee, car Consistency for Users And user preferences More devices UPnP Etc.
Brad Myers Carnegie Mellon 41
For More Information
Many papers on the Pebbles web site On PUC and other work Most programs available for free downloading:
http://www.cs.cmu.edu/~pebbles
SlideShow Commander was licensed for commercial sale Brad Myers Carnegie Mellon 42
Conclusions
As more and more electronics are computerized and communicating, mobile devices can help control them Handheld devices can
improve
the user interfaces of everything else Not just be another gadget to be learned Brad Myers Carnegie Mellon 43
Thanks to Pebbles Students
This research performed by a large number of students since 1997: Htet Htet Aung Rishi Bhatnagar Ben Bostwick Franklin Chen Yu Shan Chuang Karen Cross Carl Evankovich Marc Khadpe Dave Kong Chun-Kwok Lee Joonhwan Lee Jennifer Li Yuhua Li Leo Lie Jack Lin Kevin Litwack A. Chris Long Rob Miller Jeff Nichols Choon Hong Peck Mathilde Pignol Suporn Pongnumkul Rajesh Seenichamy Pegeen Shen Herbert Stiel Jeff Stylos Claire Tokar Marsha Tjandra Adrienne Warmack Jacob O. Wobbrock Jerry Yang Sunny Yang Brian Yeung Brad Myers Carnegie Mellon 44
Thanks to Our Sponsors!
The Pebbles research is supported by grants from: DARPA Microsoft NSF General Motors NEC Foundation Pittsburgh Digital Greenhouse And equipment grants from: Hewlett Packard Lucent Technologies Lantronix, Inc.
Lutron, Inc.
Mitsubishi Palm Computing Symbol Technologies IBM SMART Technologies VividLogic Synergy Solutions Handango
Brad Myers Carnegie Mellon 45
Human Computer Interaction Institute School of Computer Science Carnegie Mellon University
Handheld Devices for Control
Brad A. Myers
http://www.cs.cmu.edu/~pebbles
Black, followed by HAVi
Pebbles PUC Project use of the Home Audio Visual Interface
About HAVi
Home Audio Visual Interface http://www.havi.org
Protocol for controlling appliances over IEEE 1394 (i.LINK® or FireWire®) cables Mostly used for video equipment Supported by various Mitsubishi and RCA high-end products Brad Myers Carnegie Mellon 49
Our Goals for HAVi
Want to: Detect HAVi devices on Firewire bus Automatically create PUC specification from device capabilities Control all of the appliance’s features using a PC as a server (proxy) HAVi seems to have appropriate features for these Brad Myers Carnegie Mellon 50
HAVi Experience
MERL donated A 65-inch high-definition TV (WS-65909) A high definition digital VCR (HS-HD2000U) VividLogic FireBus HAVi SDK for Linux, v. 1.0.8
Brad Myers Carnegie Mellon 51
What We’ve Done
Investigated the (several) different ways of controlling HAVi devices Made simple applications to control HAVi devices But don’t have a working PUC interface Have put efforts on hold Limitations of HAVi Problems with SDK Brad Myers Carnegie Mellon 52
Television
Under the HAVi system, acts as the “master controller” Refuses to be controlled or to respond to our requests Even having the TV and PC on same bus causes problems Brad Myers Carnegie Mellon 53
VCR
Several different ways to control VCR VCR FCM “Functional Control Module” Static list of capabilities Havlet A Java program uploaded and run on TV (or PC) DDI “Data-Driven Interaction” Panel-based interface for arbitrary control Brad Myers Carnegie Mellon 54
VCR FCM
A few more features than AV/C: Play, Record, Fast Forward, Fast Reverse, Variable Forward, Variable Reverse, Stop, Record-Pause, Skip, Eject Media, Set Recording Mode, ClearRTC Get State, Get Recording Mode, Get Format, Get Position, Get capability, Get Reject Info No querying for capabilities (Apparent) Bugs in SDK on getting status Have created simple programs to control VCR using the FCM Brad Myers Carnegie Mellon 55
HavLets
Java Applets downloaded to a Java interpreter No hooks for programmatic control Very limited functions provided in the VCR’s HavLet Rewind, Play, Fast Forward, Stop, Pause, Record Status, Counter, Media Type, Recording Rate of Tape, Rate of Data Source 56 Brad Myers Carnegie Mellon
Data-Driven Interaction (DDI)
We hoped this would provide device querying and control, but… Only limited features provided in DDI Same as HavLet interface Rewind, Play, Fast Forward, Stop, Pause, Record, and some status requests Provides each UI element as a separate item E.g., “Time: ” and “54:35:33” may be two separate labels placed next to each other Need to infer relationships and types by layout Brad Myers Carnegie Mellon 57
Issues with VividLogic’s SDK
Their engineers have been very helpful SDK incomplete, poorly documented and quite buggy Only runs with a specific desktop Firewire card, only on Linux (for now) Slow to come out with new versions Needed to be “FAV” (Full AV) to use DDI but FAV support not available until Dec’02 Brad Myers Carnegie Mellon 58
Demo
Show
other
devices we have gotten to work
Brad Myers Carnegie Mellon 59
Black, followed by OLD
Adaptor for AV/C
Hand-written spec of AV/C devices PUC Java Proxy Our Java device code Our C++ code Automatic Multiple device controllers Microsoft DirectShow library AV/C protocol Firewire cable (IEEE 1394)
Brad Myers
Digital Video Camera or VCR
Carnegie Mellon
PUC Java Library GUI generation Automatic Speech UI generation
61
Adaptor using Custom Hardware for Stereo Pretends to send IR codes Reads LED panel signals to decode state Created by Pittsburgh company: Maya Design Brad Myers Carnegie Mellon 62
Driver Information Console Simulator for the Driver Information Console of Yukon Denali SUV All the modes and messages Full emulation of steering wheel buttons Brad Myers Carnegie Mellon 63
Simulator for HVAC
Brad Myers Carnegie Mellon 64