Transcript Wearable and Mobile Interfaces Professor Thad Starner Kent Lyons Contextual Computing Group GVU Center, Georgia Tech.

Wearable and Mobile Interfaces
Professor Thad Starner
Kent Lyons
Contextual Computing Group
GVU Center, Georgia Tech
What If You Could…
• Reference the world’s knowledge anytime,
anywhere?
• Be reminded of your past experiences when you
most needed them?
• Communicate with anyone, anywhere, at anytime?
• See critiques, historical information, diagnostics,
etc. overlaid in appropriate places in the physical
world?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Georgia Tech/MIT Cyborgs:
a living experiment
The Mobile Advantage
• “Augmented memory”
• “Augmented reality”
• “Intellectual advantages”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Outline
Vision
A bit of history
Human Computer Interface
Challenges of Mobility
Text entry
Displays
Gesture systems
Experiments
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Science Is Beginning to Look
Like Science Fiction
o Science fiction writers are paying attention and
provide good scenarios/motivation based on
current research!
o Synthetic Serendipity, Fast Times at Fairmont
High (Vinge)
o Historical Crisis (Kingsbury) in Far Futures
anthology (Benford)
o The Diamond Age, Snowcrash (Stephenson)
o Islands in the Net (Stirling)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Wearable Computing Vision
• Pocket or clothing based computing
– Core unit: CPU, RAM, storage, on-body network, off-body
network, and battery (Intel’s personal server)
• Runs entire day
• Wireless peripherals distributed on the body
– Choose peripherals for the task
•
•
•
•
MP3 player: +headphones
Wireless messaging: +keyboard
Email & web browsing: +display
Camcorder: +camera
– Allows rapid testing of interfaces in market
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Mobile Technology Trends
(1990-2003)
• Exponential technology
improvement
– 256X RAM
– 900X CPU
– 4000X disk
• Lagging technology
– 30X wireless speed
– 3X battery
[IEEE Computer2002]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
On-body Computing Outsells
“Personal Computers” in 2003
• 533M mobile phones
• 24M MP3 players
• 164M laptops, desktops, and servers
(Gartner, IDC)
• More e-mail sent via phone in Japan than home
PC (Newsweek)
• Motorola: “We don’t make phones, we make
wearable computers.”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Brief History
1961
1991
1966
1977
1981
1991
1992
1993
Thad Starner, GVU Center, Georgia Tech
1986
1993
1996
ISWC2004
Miniature Head-up Displays
MicroOptical prescription
display eyeglasses
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Keyboards
• Twiddler
–
–
–
–
Chording
In 5 min. alphabet
In 1 hr touch typing
Speeds >60 wpm
• Embroider it in a
jacket!
Thad Starner, GVU Center, Georgia Tech
ISWC2004
CharmIT Wearable Computer
266MHz Intel Pentium or 800MHz Transmeta Crusoe
Thad Starner, GVU Center, Georgia Tech
(www.charmed.com)
ISWC2004
Quick Survey
• How many people
– Own one?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Quick Survey
• How many people
– Own one?
– Have it with you?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Quick Survey
• How many people
– Own one?
– Have it with them?
• On average in my talks
– ~50% own
– ~25-50% of owners have it with them
• Many problems - interface is key
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Why not a PDA?
• Too much cognitive load
– Augment, not replace task
– Two hands, both eyes
• Socially awkward
• Low functionality
–
–
–
–
Input speed
Data storage
“Hot sync” effect
Applications
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Vertical Markets
•
•
•
•
•
•
•
Warehouse picking
Inspection
Maintenance
Repair
“Line-busting”
Security
Military (Land Warrior/Pacific Consultants)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
What is HCI?
The study of people and computing
technology and the way they influence each
other
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Why study HCI?
We are surrounded by unusable and
ineffective systems!
It’s not the user’s fault!
Costs are high in $$, time, frustration, missed
opportunities
“Folk HCI” isn’t the answer
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Concepts, Principles, Guidelines
• No “cookbooks”
• No simple, universal checklists
• There are many concepts, principles, and
guidelines
• Understand the higher level principles that
apply across situations, display types, etc.
• Implement the standards and guidelines
Thad Starner, GVU Center, Georgia Tech
ISWC2004
UI Design Principles (Dix et al.)
•
Categories
1. Learnability
•
support for learning for users of all levels
2. Flexibility
•
support for multiple ways of doing tasks
3. Robustness
•
support for recovery
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Frameworks for HCI
• “Folk HCI”
• Humans as sensory processors
– Human factors, experimental psych
• Humans as interpreters/predictors
– Cognitive psych, AI
• Humans as actors in environment
– Activity theory, ethnography, ecol psych
Thad Starner, GVU Center, Georgia Tech
ISWC2004
What makes a system “usable”
• Humans as sensory processors
– Fit with human limits
• Humans are interpreters/predictors
– Fit with knowledge and task
• Humans as actors
– Fit with task environment and social context
Thad Starner, GVU Center, Georgia Tech
ISWC2004
HCI Methods
• Humans as sensory processors
– Quantitative evaluation
• Humans are interpreters/predictors
– Task analysis, Cognitive walkthrough
• Humans as actors
– Ethnographic field work, participatory design
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Cognitive Frameworks
• Model-Human Processor (Card, Moran and
Newell)
• Situated Action (Suchman)
• Activity Theory (Vygotsky, Nardi)
• Distributed Cognition (Hutchins)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Two view of interaction
• Interaction with
– Software as a tool or machine
– Interface is usability-engineered membrane
– Human as processor & interpreter models
• Interaction through
– Software as medium to interact with task objects or people
– Interface plays a role in social context
– Human as interpreter & actor models
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm shifts
Networks and time-sharing
Video display units
Programming toolkits
Personal computing
Windows
Metaphors
# People/# Computers
Thad Starner, GVU Center, Georgia Tech
Direct manipulation
Language vs. action
(agents)
Hypertext/WWW
Multi-modality
CSCW
Ubiquitous computing
In/Out
Tech
Style
ISWC2004
Paradigm Shifter: Vannevar Bush
• “As We May Think” - 1945 Atlantic Monthly
“…publication has been extended far beyond our
present ability to make real use of the record.”
• Postulated Memex device
– Stores all records/articles/communications
– Items retrieved by indexing, keywords, cross references (now called
hyperlinks)
– (Envisioned as microfilm, not computer)
• Interactive and nonlinear components are key
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm Shifter: J.R. Licklider
• 1960 - Postulated
“man-computer symbiosis”
• Couple human brains
and computing machines
tightly to revolutionize
information handling
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm Shifter: Ivan Sutherland
• SketchPad - ‘63 PhD thesis at MIT
–
–
–
–
–
–
–
Hierarchy - pictures & subpictures
Master picture with instances (ie, OOP)
Constraints
Icons
Copying
Light pen for input
Recursive operations
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm Shifter: Douglas Engelbart
• Landmark system/demo:
–
–
–
–
–
–
Mouse, windows
Hypertext
Multimedia
High-res display,
Shared files, CSCW,
Electronic messaging, teleconferencing, ...
• Inventor of mouse and a chording keyboard
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm Shifter: Alan Kay
• “Personal Computing”
• Dynabook: Notebook sized
computer loaded with
multimedia and can
store everything
• Desktop interface metaphor
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Paradigm Shifter: Ted Nelson
• Computers can help
people, not just business
• Coined term “hypertext”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
WIMP
• Windows, Icons, Menus, Pointers
• Timesharing=multi-users; now we need
multitasking
• WIMP interface allows you to do several
things simultaneously
• Has become the familiar GUI interface
• Xerox Alto, Star; early Apples
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Direct Manipulation
• ‘82 Shneiderman describes appeal of
graphically-based interaction
–
–
–
–
–
object visibility
incremental action and rapid feedback
reversibility encourages exploration
replace language with action
syntactic correctness of all actions
• WYSIWYG, Apple Mac
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Metaphor
• All use is problem-solving or learning to
some extent
• Relating computing to real-world activity is
effective learning mechanism
– File management on office desktop
– Financial analysis as spreadsheets
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Speech, Language?
• Actions do not always speak louder than
words
• Interface as mediator or agent
• Language paradigm
• How good does it need to be?
– “Tricks”, vocabulary, domains
• How “human” do we want it to be?
– (HAL, Bob, PaperClip)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Multimodality
• Mode is a human
communciation channel
– Not just the senses
e.g., speech and non-speech audio
are two modes
• Emphasis on simultaneous
use of multiple channels for I/O
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Hypertext
• Think of information not as linear flow but as
interconnected nodes
• Nelson’s hypertext
• Bush’s MEMEX
• Non-linear browsing
• WWW ‘93
• Hypermedia
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The Interconnected Web
• The Network is the Computer
e.g. seti@home
Thad Starner, GVU Center, Georgia Tech
ISWC2004
CSCW
•
•
•
•
Computer-Supported Cooperative Work
No longer single user/single system
Social aspects are crucial (micro-social)
E-mail as prominent success but other
groupware still not widely used
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Ubiquity
• Person is no longer user of virtual device
but occupant of virtual, computationallyrich environment
• Can no longer neglect macro-social aspects
• Late ‘90s - PDAs, VEs, …
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Copy Experiment
•
•
•
•
•
•
Between-subjects design
Pair-up (second hands?)
Task 1: Copy “Thanks to” page
Task 2: Copy “Ubiquity” page
Two courses (A/B)
NASA TLX
• [vest video; experiments in the field]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Controlled Studies
• CMU VuMan3 (Siewiorek/Smailagic)
–
–
–
–
Military inspection task
2:1 savings in personnel
40% faster
Custom design (many design generations)
• Georgia Tech Task Guidance (Ockerman)
– Small airplane inspection by pilots
– Basic manual emulation– no feedback
– Wearable interface hindered expert!
• Similar to checklist?
• Providing context helped
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Symbol Technologies WS series
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Symbol’s Success
• $5 million development costs
–
–
–
–
People sweat
Body armor
Plastic wears
Wearer buy-in through demonstration
• > 100,000 units; $3500-$5000 list
• Unique differentiator
• New markets
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Speed of Access Experiment
• Pair up
• Person A: timer
• Person B: subject?
• Show of hands histogram
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Application: Calendaring
• One of the most common PDA applications
• One of the most desired functions
• Occurs routinely in social conversation
– One-on-one
– Conferences
– Meetings
• Anecdotal observation of dissatisfaction
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Scheduling Device Survey
• 138 subjects
– Georgia Tech student center
– 90% students; 88% age 18-25; 70% male
• Survey
– What is your primary scheduling system while mobile?
– 8 Likert scale questions on effectiveness, ease of use, speed, and
reliability
– Open response questions
• Schedule four appointments
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Appointment Tasks
• “Could we meet sometime next Monday?”
• “Could we schedule a time to meet in the
second week of February?” (three months in
future)
• “Could we schedule a time to meet
tomorrow?”
• “Could we reschedule our appointment in
February from the 10th to the 11th?”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Videotaped Interactions
Scheduling device
Subject view
Timed retrieval, navigation, and entry
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Actual vs. Claimed Usage
Memory
Scrap
Planner
PDA
Other
# Used
Memory
24
9
16
4
4
57
Scrap
1
13
13
1
4
32
14
1
1
16
Planner
PDA
8
Other
1
#Claimed
26
1
22
Thad Starner, GVU Center, Georgia Tech
44
14
8
6
8
15
121
ISWC2004
Actual vs. Claimed Usage
Memory
Scrap
Planner
PDA
Other
# Used
Memory
24
9
16
4
4
57
Scrap
1
13
13
1
4
32
14
1
1
16
Planner
PDA
8
Other
1
#Claimed
26
1
22
Thad Starner, GVU Center, Georgia Tech
44
14
8
6
8
15
121
ISWC2004
Actual vs. Claimed Usage
Memory
Scrap
Planner
PDA
Other
# Used
Memory
24
9
16
4
4
57
Scrap
1
13
13
1
4
32
14
1
1
16
Planner
PDA
8
Other
1
#Claimed
26
1
22
Thad Starner, GVU Center, Georgia Tech
44
14
8
6
8
15
121
ISWC2004
Actual vs. Claimed Usage
Memory
Scrap
Planner
PDA
Other
# Used
Memory
24
9
16
4
4
57
Scrap
1
13
13
1
4
32
14
1
1
16
Planner
PDA
8
Other
1
#Claimed
26
1
22
Thad Starner, GVU Center, Georgia Tech
44
14
8
6
8
15
121
ISWC2004
Abandonment rates
• 43% of PDA users switched
• 68% of planner users switched
• Memory and scrap paper dominated
• Hypothesis: Users switch to mechanisms
that are faster to access (similar to
Miller68?)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Timing (in seconds)
Device Retrv.
Scrap
Navig. Retrv.+ Entry
Navig.
Total
17.8
18.1
35.9
17.8
Planner 11.8
7.6
19.4
12.5
31.9
PDA
12.7
23.7
14.0
37.7
11.0
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Actual vs. Claimed Usage
Memory
Scrap
Planner
PDA
Other
# Used
Memory
24
9
16
4
4
57
Scrap
1
13
13
1
4
32
14
1
1
16
Planner
PDA
8
Other
1
#Claimed
26
1
22
Thad Starner, GVU Center, Georgia Tech
44
14
8
6
8
15
121
ISWC2004
Abandonment by Task
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Buffering Information
for Later Entry
• 42% of scrap and memory users said they
would later enter appointment in another
system for first (“next week”) appointment
• 86% for second (3 months away)
appointment
• Reduce cognitive load? Social effect?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Summary
• Overall usage speed is similar between methods
• Users tend to switch to faster access systems when
scheduling appointments when mobile
• Appointments buffered for later entry
– Even though 2X in overhead (waste a minute later to save a second
during conversation)
– Transfer to more formal devices for more distant events
• Future work: Larger study on office workers
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Just-in-time Information Retrieval
• Automatically provide information
• Based on local environment
• Do it without driving people nuts
Speech Agent Experiment
• What if you wanted a speech-enabled agent
running all the time?
• What would you do with it?
• How would you implement it?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The Jane Experiment
• Continuous audio-based agent
– Inspired by Card’s “Ender’s Game”
– Access to user’s e-mail
– Internet search engines
• Wizard of Oz experiment failed:
–
–
–
–
“Agent” could not respond quickly enough
Audio output was interruptive
Not enough context to be pro-active
Context could not accumulate due to experimental
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Vocollect Series
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Nomadic Radio
• Audio interface
– Voicemail, news, email
• Dynamic interruption
– Importance of info
– Personal profile
– Conversation detection
(Sawhney, MIT Media Lab)
Example of Existing Interaction:
A Hallway Conversation
Alice: Bob, can we meet next week?
Bob pulls out his PDA.
Bob:
Next week you said?
Bob starts the scheduling application
Alice: Yes, how about Monday?
Bob uses the stylus to switch to month
view.
Bob:
Monday? Let me check.
He selects Monday to change
to day view.
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Example of Existing Interaction:
A Hallway Conversation
Bob:
I'm busy all day Monday
Bob advances the calendar one day.
How about Tuesday?
Alice: Tuesday at one then?
Bob selects the 1:00pm entry.
Bob:
Sounds good. I'll pencil you in at one.
Bob enters Alice's name at 1:00 and
puts away his PDA.
Conversations are short:
Minimize time with PDA
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The Calendar Navigator Agent
• Navigates calendar based on speech
• Push-to-talk and key phrases
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The CNA
Alice:
Bob, can we meet next week?
Bob starts CNA
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The CNA
Bob: Next week you said?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The CNA
Alice:
Yes, how about Monday?
Bob: Monday? Let me check.
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The CNA
Bob: I'm busy all day Monday.
How about Tuesday?
Thad Starner, GVU Center, Georgia Tech
ISWC2004
The CNA
Alice: Tuesday at one then?
Bob: Sounds good.
I'll pencil you in at one.
Thad Starner, GVU Center, Georgia Tech
ISWC2004
DialogTabs
• Short term audio reminder
• Tabs for speech segments
• Random access to audio with transcript
Thad Starner, GVU Center, Georgia Tech
ISWC2004
DialogTabs
Eve:
Bob, please call our client about the
proposal. They are out of the office;
the number is 555-1292.
Bob:
555-1292. I'll call them now.
• Bob confirms the number and Eve knows he
heard it correctly.
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Speech Courier
• Relay conversation to third party
• Eve: “Alice will email you the write-up for
our new proposal”
– Bob understands he will get an email
– Alice knows to send the email
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Remembrance Agent
Margin Notes
Prosthesis For The Brain
• Less autonomy
• Constant, low-load
communication
• Tight integration
with environment
and task
JITIR Interfaces
• Progressive disclosure (Ramping interface)
– Low-cost false positives
– Lots of opportunities to bail out
– Allow control over when information is viewed
• Follow proximity compatibility principle
– Use local environment as part of interface
• Two-second rule (Miller, 1968)
Controlled Evaluation Results
• Essay-writing experiment with news articles
• Subjects read three times as many articles
using the RA as with a search engine
• RA use was in addition to search engine
use, not a replacement
Ease-of-access changed behavior
Value of JITIRs
• Provide new material
– Answering questions as they’re asked
• Provide supporting material
– “I write opinions, the RA gives me the facts”
• Contextualize
– “…nice to see how other people talked about
this.”
• Help with another (related) task
Text Entry Experiment
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Everyday Applications
•
•
•
•
Consumer devices (CD, movies, cell)
Instant messenger (zephyr)
Instant reference (webster, google)
Remote monitoring (telnet)
video
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Text Entry
• Mobile phones (multi-tap, T9)
• Virtual keyboards (ATOMIK, SHARK2)
• Chording systems (Twiddler, ChordTap)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Twiddler
• Mobile, one-handed
• 3 x 4 grid of buttons
• Chording
fga
e
cd
m
ijklb
xyzh
vw
rstu
q
p
o
n
CHI 04 (Ch. 4)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Comparison to Desktop Rates
•
•
•
•
•
•
•
•
World record QWERTY (GBWR98): 192 wpm
Fastest Twiddler burst:
130 wpm
Highly skilled secretary:
70-90 wpm
Fastest Twiddler user average:
70 wpm
Highest U.S. pay grade for typists:
50 wpm
Average Twiddler rate:
47 wpm
Goal of semester typing class:
40 wpm
Average desktop rate:
30 wpm
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Extended Avg. Learning Rates
67.1
45.2
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Errors
• Errors equivalent to
other mobile text
entry methods
• More errors as
optimized $ payoff
(wpm * accuracy *
$1)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
“Blind” typing
• Users the same speed or slightly faster without
visual feedback (p <0.05)
• Same or less errors (p<0.05)
• Market: teenagers messaging in class!
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Multiple Character Chords
(MCCs)
• A MCC is significantly
faster to type than
individual letters (e.g.
“the_” vs. “t” “h” “e”
“_”)
• MCCs did not occur
frequently enough to
effect typing rates
• Still testing
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Display Experiment
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Information Where You Need It
Factory
Warehouse
(Xybernaut)
Georgia Tech
BattleField
(Motorola)
Columbia University
Augmented Reality (1993)
• Applications
– Instruction
– Mobile information
• Focus on graphics, speed
• Good evaluation
• Wired ultrasonic sensors
http://www.cs.columbia.edu/graphics/
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Vision-based AR: finger as
mouse (1995)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Repair/Inspection/Maintenance
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Augmented Reality
Wiring AR System
Billiards Assistant
(Mizell, Boeing)
(Jebara, MIT Media Lab)
Physical World Wide Web
Thad Starner, GVU Center, Georgia Tech
ISWC2004
ARToolkit (Billinghurst)
Thad Starner, GVU Center, Georgia Tech
video
ISWC2004
Mobile Gesture Systems
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Gesture Pendant
• Command gestures for
controlling appliances in the
home
• Small “lexicon”
– Volume up/down
– Channel up/down
– Fast forward/rewind
• “Push to gesture”
• Aim at device to control
[video]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Near-Infrared Computer Vision
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Prototype Operation
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Successes and Challenges
• Demonstrated to ~500 users
(ACM01, Chicago Museum,
etc.)
• Intuitive, easy to control
• Tremor monitoring for
Parkinson’s disease
• Requires visual feedback
• Sunlight/bright lights interfere
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Gesture Panel: Low Attention
Controls for Automobiles
• Distraction from
internal controls
causes accidents
• Visual attention
important
• Extreme lighting
– Tunnels
– Sunlight
Thad Starner, GVU Center, Georgia Tech
4%
3% 2%
5%
Standard Radio
Instrument Cluster
6%
Gearshift
HVAC
7%
52%
Wiper/Washer
Cell Phone
2 Way Radio
8%
Pedals & Floor
Unspec & Misc
13%
ISWC2004
Gesture Panel Operation
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Gesture Panel
• Small lexicon of gestures
• Can use context to determine action
• No moving parts
• Works in any lighting
• Prototypes in laboratories at Visteon
(formally Ford Electronics) and
DaimlerChrysler
Thad Starner, GVU Center, Georgia Tech
ISWC2004
FreeDigiter
• Problem: portable electronics becoming too small to
control by buttons
– Hearing aids
– MP3 players
– Bluetooth mobile phone headsets
• Other situations where buttons not appropriate
– Driving (too much attention or hand-eye coordination)
– Messy hands (cooking, surgery, car mechanic, etc.)
[demo]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
BlinkI: Controlling Appliances
with Eyeblinks
•
•
•
•
30,000 paralyzed “locked-in” patients
ALS “Lou Gehrig’s” disease
Can control only eyes
Communication and control of appliances
very slow
• Idea: Blink the cadences to songs to control
devices
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Examples
• Songs encode number and rhythm of blinks
[video]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
BlinkI Results
•
•
•
•
Dynamic Time Warping
99% accuracy over 10 songs
303 examples
Leave one out testing
• Currently installing a version in the Aware
Home
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Other Applications
•
•
•
•
•
•
Cameras
Binoculars
Night vision goggles
Automobiles
Low vision systems
Security
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Prescott: Eyeblinks for Security
• Complement face recognition with a
blinking pattern as a passcode (song
cadence, Morse code, etc.)
• Blinking helps locate the face
• Blinking is hard to observe without the
user’s knowledge
• Hands-free and silent
Thad Starner, GVU Center, Georgia Tech
ISWC2004
“Blinkprints”
• How to defeat Prescott
– Make a mask of the user’s face
– Determine what song they are blinking
• Surprise!: people differ in how they blink
the same pattern
• Blinkprints: new biometric?
• Similar to speaker id in speech
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Prescott Results
• 82% accuracy with 9 subjects blinking the same
pattern
• Features: average optical flow direction and
length of pattern (!?)
• Why does it work?
– Deformation of eyelid over eye?
– Eye rolling when blinking?
• Suitable for verification (face recognition +
blinked passcode + blinkprint)
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Recognizing Gesture
• Wearable American Sign Language
recognition: 97% accuracy
Thad Starner, GVU Center, Georgia Tech
ISWC2004
ASL->English One-Way
Translator
• Inspired by “Phraselator” used by medical doctors in Iraq
– PDA that translates English to Arabic
– User speaks; PDA presents top 5 phrases; user selects
– Phrases and questions are designed to be answered by head nods
(yes/no), pointing, or numbers
• “Nod your head up and down if you are hurt”
• “Point to where you are hurt”
• “How many days ago were you hurt”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Phraselator
• “Raise your
hand if you
understand
me”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Need for Simple ASL Translators
•
•
•
•
Interpreters cost $80/hour
Inconvenient to schedule; not spontaneous
English is a second language to the Deaf
Domains:
–
–
–
–
–
Doctor’s office (privacy)
Lawyer’s office (privacy)
Airport directions (convenience)
Car accident (convenience)
Apartment hunting
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Apartment Hunting Domain
• Interpreters too expensive and too
inconvenient to take to every appointment
• Mostly simple questions and responses
– “Can I have a cat?”
– “How many bathrooms?”
– “Where is the bathroom?”
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Head Mounted Camera and
Accelerometers
Bluetooth 3-axis accelerometers
60 hours battery life
[videos]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Acceleglove
• Phrase level
recognition
• 141 signs
• 94% word
accuracy
• Next step:
wireless rings
[FG2004]
Thad Starner, GVU Center, Georgia Tech
ISWC2004
“Take Home” Points
• Applying HCI to mobile systems
• Computer is not the focus!
– Attention managing; socially appropriate
• Tasks: Augmenting memory,
communication, located information, ???
• First steps
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Resources
• Human Computer Interaction (second ed.) Dix,
Finlay, Abowd, and Beale
• UIST, CHI, Mobile HCI, ISWC, Ubicomp,
CSCW, InfoVis
• IEEE Wearable Information Systems Technical
Committee (computer.org)
• www.cc.gatech.edu/ccg
• Research mailing list: wearables[at]cc.gatech.edu
• Kent[at]cc.gatech.edu
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Sponsors
IEEE Computer Society/ACM/ISWC
National Science Foundation
NIDRR Wireless RERC
Thad Starner, GVU Center, Georgia Tech
ISWC2004
Thanks to
Beth Mynatt & Bradley Rhodes
Georgia Tech, MIT, CMU, U. Washington, U.
Oregon, Columbia, ETH, UMIT
Symbol, IBM, Vocollect
Thad Starner, GVU Center, Georgia Tech
ISWC2004