Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Using Hand-Held Computers and PCs Together: The Pebbles Project Brad A.
Download
Report
Transcript Human Computer Interaction Institute School of Computer Science Carnegie Mellon University Using Hand-Held Computers and PCs Together: The Pebbles Project Brad A.
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Using Hand-Held Computers
and PCs Together:
The Pebbles Project
Brad A. Myers
[email protected]
http://www.cs.cmu.edu/~pebbles
Multiple Devices
Most of our time spent in places where there
is embedded technology
Offices, meeting rooms, classrooms, homes
Often multiple devices will be available:
Mobile phone and PDA and a PC
Multiple people’s PDAs
PDAs in a “Smart Room”
Brad Myers
2
Handhelds will be communicating
802.11
BlueTooth
Cell-phone network
(Infrared)
Brad Myers
3
Premises of our Research
“With the coming wireless technologies,
connecting the PCs and PDAs together
will no longer be an occasional event
for synchronization. Instead, the
devices will frequently be in
close, interactive communication.”
—
Brad Myers, “Using Hand-Held Devices and
PCs Together,” Comm. ACM,
Vol. 44, No. 11. Nov., 2001. pp. 34 - 41.
Brad Myers
4
Research Agenda
How can multiple devices
be used effectively together,
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
Brad Myers
5
Research Agenda
How
can
multiple
devices
“Multi-Machine
be used effectively together,
User
Interfaces”
at the same time?
How can the user interface
and functionality be spread
across multiple devices?
Brad Myers
6
Example: Power Point Control
Use PC to give the presentation
Use hand-held to control the PC
Two-way communication
Hand-held shows picture of slide,
notes, list of titles, timer, etc.
Brad Myers
7
Pebbles is:
P DAs for
E ntry of
B oth
B ytes and
L ocations from
E xternal
S ources.
http://www.pebbles.hcii.cmu.edu/
Brad Myers
8
Handhelds in an Office
Augment desktop applications
Use multiple devices at the same time
Brad Myers
9
Handhelds in Meetings
Attendees use handhelds to interact and
annotate presentation
Augment collaboration
Take notes
Brad Myers
10
Handhelds in Classrooms
Data projectors for instructor’s slides
Students could have computers for:
Notetaking linked to instructor’s slides
In-class testing
Running simulations
Brad Myers
11
Handhelds in Homes
Interact with embedded computation
“Smart homes”
Not just speech and vision as interfaces
Interact with appliances, lights, etc.
“Personal Universal Controller” (PUC)
Brad Myers
12
Office Use (For Individuals)
How can handhelds augment desktop
applications?
Applications for Individuals
Extra input and output devices have been
shown to be useful
But can be expensive and hard to configure
People have PDAs and are attached to PC
For example, cradles for recharging
Customizable, extensible
Extend desktop applications
Brad Myers
14
Scrolling with the PDA
For scrolling using the non-dominant hand
Studies showed parallel and efficient uses of
both hands together
Generates Windows scrolling events
Brad Myers
15
Results of Study of Scrolling with PDA
Scrolling with buttons on PDA was fastest
PDA scrollers similar to mouse speed
Using 2 hands is effective!
Time to Scroll 10 Pages
Sec
100
80
60
40
20
Winner!
0
Trial 2
ButtonScroller
Mouse
AbsScroller
Trial 3
SlideScroller
Scroll Wheel
RateScroller
Brad Myers
16
Remote Clipboard
Transfer information between PDA and PC
Connects their clipboards together
Transfer content or reference
Works with all applications
Also between multiple
computers
Brad Myers
17
Shortcutter
User-created panels of controls
Create custom interfaces and
extensions to PC applications
And then take them with you
Direct manipulation
for edit, then set
properties
Palm or PocketPC
Brad Myers
18
Shortcutter Widgets
Buttons
Sliders
Knobs
Mouse pad
Graffiti Pad (Palm)
Gesture panel
Brad Myers
19
Shortcutter Actions
Send any keyboard key, mouse button,
scrolling action or string to PC
Open a file or URL
Run an application
Invoke any PC menu or button
Windows message
Recorded
Switch to a different Shortcutter panel
Control the Mouse
Brad Myers
20
Shortcutter Actions, cont.
Control external devices
through PC’s serial port
Macro
Directly (e.g., projectors)
X-10 for electrical devices
Can be multi-application
Application-specific
Same button, different messages
Useful for application sets:
browsers, compilers
Brad Myers
21
More Scenarios of Use
Lean-back mail reading
Controlling WinAmp
… and many others
Brad Myers
22
Study of Individual Use
Time to tap on button depended on size
Few errors
People often didn’t look at PDA
Tap Tim e
(m sec)
Error Rate (%)
Button Task: Tap Time
100
1000
903.68
900
800
700
Button Task: Error Rate
684.74
702.55
683.01
714.43
742.25
779.20
793.82
90
80
70
600
60
500
50
400
40
300
30
200
20
100
10
=5
48
)
0.70
1.05
1.57
2.61
2.27
1.74
2.11
2x2
2x3
3x2
2x4
4x2
3x4
4x3
4.87
0
4x4
4x
4
(n
=5
56
)
4x
3
(n
=5
66
)
3x
4
(n
=5
60
)
4x
2
(n
=5
59
)
2x
4
(n
=5
64
)
3x
2
2x
3
2x
2
Brad Myers
(n
=5
65
)
(n
(n
=5
67
)
0
23
Study of Individual Use
Moving hands to both PDA and mouse
only about 15% slower than just moving to
the mouse
msec
1H Keyboard->Mouse
1H Keyboard->PDA
1H Mouse->Keyboard
1H PDA->Keyboard
Keyboard -> Mouse&PDA
728
744
701
639
838
Mouse&PDA -> Keyboard
791
% slower
15.1%
12.8%
Brad Myers
24
Use in Meetings
Enhance group’s collaboration and control
Original Application:
Remote Commander
Allow PDAs to control a PC
Can be used with any application
Uses the standard (single) cursor
Don’t have to jump up and grab
mouse
Perform all mouse and
keyboard functions
Use PDA like touchpad
Graffiti or our own
pop-up keyboard
Brad Myers
26
PocketPC version
Get PC’s screen onto PocketPC
Full view, or one-to-one zooming
Scroll with iPaq’s buttons
Brad Myers
27
Scribble
Multiple people draw on top of whatever on
PC screen, not just PowerPoint
Each user has own cursor
and color
Save by PrintScreen
Erase by refresh
Brad Myers
28
SlideShow Commander
For PowerPoint
Use PC to give the presentation
Full features of PowerPoint
Use hand-held as “remote control” for PC
PC Only: Windows 95, 98, NT, 2000, ME, XP
Can wander away from the keyboard
Two-way communication
View, Navigate, Highlight, Time
Brad Myers
29
View on Hand-Held: slide
See thumb-nail of current slide
Black and white or color
Brad Myers
30
View on Hand-Held: notes
See the notes of the current slide
Brad Myers
31
View on Hand-Held: titles
See the list of titles
Brad Myers
32
View on handheld: Time
Multi-function timer
View large
And in corner
Brad Myers
33
Control Demonstrations on Palm
Control PC applications and external devices
Uses Shortcutter
Easy demo and resume show
Brad Myers
34
Control Demonstrations on PocketPC
Task list of PC applications
Tap brings one to front
PowerPoint continues
behind – easy to resume
Brad Myers
35
Commercialized
SlideShow Commander
commercialized by:
Synergy Solutions, Inc.
http://www.synsolutions.com/
Available at Office Depot,
CompUSA, etc.
Palm and PocketPC
http://www.slideshowcommander.com/
Brad Myers
36
Studies of Presentations
Summer study of 2 HCI Master’s students
Contextual Inquiry of 9 presentations
Found 220 “breakdowns”
Audience: 10 to hundreds
7 used PowerPoint, 4 used NetMeeting
Most were minor problems
Averaged 8.7 lost minutes per talk (14.5%)
Designed SlideShow Commander to
eliminate some of these problems
Brad Myers
37
MultiCursor
For special applications that are aware of
multiple inputs
E.g.: Shared Whiteboard
Single Display Groupware
Multiple people, one display
A number of interesting
issues:
Palettes, widgets
Section handles
Undo
Brad Myers
38
Chat
Communicate to another PDA user through
the PC
PC serves as a conduit
For side notes and messages
For example, in negotiation
meetings
Send to all or to a specific
person
Brad Myers
39
Command Post of the Future
Large DARPA funded project
Make commanders more effective
Brad Myers
40
Study of Laser Pointers
Studied properties of laser
pointer interaction techniques
Hand-wiggle 8 pixels
Delay until moving average stable 1.5 sec
Delay until target acquired 1.5 sec
Different devices and poses do not help much
Brad Myers
41
Tap Speed Study
Tapping directly on
SmartBoard is fastest
Semantic Snarfing next
But high error rate
Then regular mouse
Laser pointing slowest
Used separate physical
button on handheld
1000
9.00%
8.00%
7.00%
6.00%
5.00%
4.00%
3.00%
2.00%
1.00%
0.00%
800
600
400
200
0
Mouse SmartBoard Laser
Pointer
Movement Time (ms)
Semantic
Snarfing
Error Rate
Brad Myers
42
“Semantic Snarfing”
Interacting at a distance
Grab contents to handheld
Picture
Menus
Text
Re-visualization
“Magic Lenses”
Brad Myers
43
Classroom Use
Using handheld computers in classrooms
For testing
Improve large lectures
Provide immediate feedback to instructor
Hardware
HP donated 110 Jornada 680 and 100 720
computers
Windows CE
Lucent donated Wavelan wireless cards
Brad Myers
45
Context
Collaborator: Prof. David Yaron of Chemistry
Chemistry 09-106: Modern Chemistry II
Spring, 2000 and Spring, 2001
About 90 students
Mostly freshmen
Loaned one Jornada and Wavelan card to
each student for the whole semester
So would get used to using it
Offset technical difficulties with benefits to
student
Brad Myers
46
Wireless Andrew
CMU has almost complete coverage of
802.11b support
Wavelan
Can use computers wirelessly in all
classrooms and offices
Donations from Lucent, etc.
Funding from state of Pennsylvania, etc.
Most lawns, dorm rooms, etc.
Windows, Mac, Linux, Windows CE support
Brad Myers
47
Pebbles Tests
Create through html pages
Arbitrary html in questions
and answers
Pictures, formatting, links
Use FrontPage, etc. to
author
Embed tags to show
question and answers
Brad Myers
48
Concept Test
Students allowed to answer multiple times
Server stores all answers
Prof. Yaron displayed
questions and answers on
the board
Evidence that significantly
contributes to learning
—
Mazur, E. (1997).
Peer Instruction:
A User's Manual, Prentice Hall.
Brad Myers
49
Instructor’s View
In class, instructor sees a chart of answers
Optionally refreshes every 3 seconds
May be
projected for
whole class to
see
Top displays
instructions for
students
Brad Myers
50
Results
Unfortunately, no data (yet) about effects on
learning
Students thought concept tests were a
valuable part of the class
Students significantly preferred using
handhelds for concept tests.
Raise Hands Hold up cards Handhelds
25%
4%
63%
Brad Myers
51
Home Use
Personal Universal Controller
Help for people with muscular disabilities
Personal Universal Controller
Two-way communication
Appliances describe their functions
Handheld PUC creates interface based on descriptions
Handheld PUC controls the appliance
Appliance sends back status as feedback
Specifications
Control
Feedback of Status
Brad Myers
53
Initial Experiments
Use real PocketPC
Pretend that controls devices
Brad Myers
54
PocketPC results
For both appliances, users of actual interfaces:
Took about twice as long
Made at least twice as many mistakes as users of
the handheld interfaces
Needed external help five times more often
Brad Myers
55
Current Steps
XML specification language and
protocol for describing appliances
Create panels automatically from
the specification
Find real
appliances which
we can control
Specification
Language
<?xml version="1.0" encoding="UTF-8"?>
<spec xmlns="puc.xsd"
xmlns:xsi="http://www.w3.org/2001/XMLSc
hema-instance"
xmlns:xsd="http://www.w3.org/2001/XMLS
chema"
name="Audiophase 5 CD Stereo">
<groupings>
<state name="PowerState">
<type name="OnOffType>
<valueSpace>
<boolean/>
</valueSpace>
<valueLabels>
<map value="false">
<label>Off</label>
</map>
<map value="true">
<label>On</label>
</map>
</valueLabels>
</type>
<labels>
<label>Stereo Power</label>
<label>Power</label>
<label>Powr</label>
<label>Pwr</label>
</labels>
<priority>10</priority>
</state>
Brad Myers
56
Current Work: Handhelds for
People with Muscular Disabilities
Using handhelds as interface to PCs
People with Muscular Dystrophy
have fine-motor control but lose
gross motor control
Difficulties with mouse and keyboard,
but stylus OK
Handhelds as interface to other devices
Control room lights, telephone, wheelchair, etc.
Brad Myers
57
Remote Commander Adaptations
Faster Acceleration
More flexible tapping
Turn off key repeat
Multiple Keyboards
Brad Myers
58
General Architecture
Brad’s
Palm Pilot
On the PC
Rob’s
Pocket PC
Ben’s
Windows CE
Windows
event
stream
PebblesPC
Serial,
IR or
sockets
Various PDA
apps
Any PC app.
RemoteCmd
MultiCursor
Direct
connection
or sockets
Slideshow
Commander
PebblesDraw
PowerPoint
OLE
Automation
Various dlls
Brad Myers
59
Protocols
Can use Pebbles protocols to develop your
own application
Use by:
Libraries for Palm, Windows CE, PC
Independence from communication medium
PalmAmp from IronCreek Software
Intel research
others…
Windows messages or sockets
Brad Myers
60
Downloads
Most of this software is available for free
downloading
First release in Feb, 1998
Just released version 5 (Feb, 2002)
Downloaded over 30,000 times
About 200 times a week
http://www.pebbles.hcii.cmu.edu/
Brad Myers
61
Conclusions
Handhelds are becoming ubiquitous
Connecting technologies will improve
Important to study how can be used when
connected to computers and each other
“Multi-Machine User Interfaces” will be
increasingly important
People will want to use the most convenient
device for their information and control
Brad Myers
62
Thanks to Our Sponsors!
Supported by grants from:
DARPA
Microsoft
Pittsburgh Digital Greenhouse
NSF
And equipment grants from:
Hewlett Packard
Lucent Technologies
Palm Computing
Symbol Technologies
IBM
SMART Technologies, Inc.
Synergy Solutions, Inc.
Handango
Brad Myers
63
Human Computer Interaction Institute
School of Computer Science
Carnegie Mellon University
Using Hand-Held Computers
and PCs Together:
The Pebbles Project
Brad A. Myers
[email protected]
http://www.cs.cmu.edu/~pebbles