Transcript Software Development Forum Emerging Technology SIG Weds. Sept 9 2009 Augmented Reality Clark Dodsworth Osage Associates Maribeth Back FX Palo Alto Laboratory [email protected] [email protected].

Software Development Forum
Emerging Technology SIG
Weds. Sept 9 2009
Augmented Reality
Clark Dodsworth
Osage Associates
Maribeth Back
FX Palo Alto Laboratory
[email protected]
[email protected]
Tonight’s Plan
Theoretical foundations
Scope
My background
Brief AR history
Directions for development, revenue + evolution
Maribeth Back, FX-PAL:
“High Tech Chocolate: Exploring Mixed Reality applications for Industry”
Ribbit.com demo by Kristi Wells and Brendan Lee
Augmenting Reality and Revenue
Connecting Mobile, Sensors, Location & Layers
Clark Dodsworth
[email protected]
Alfred North Whitehead,
first augmented reality theorist:
#1: “Civilization advances by extending the number of important
operations which we can perform without thinking of them.”
...effective AR doesn’t increase the cognitive load
#2: We think in generalities, but we live in detail.”
...AR tasks: managing detail
#3: “The art of progress is to preserve order amid change and to
preserve change amid order.”
...contextually dynamic data management & display via AR
Scope - General
AR = subset of Ubiquitous Computing: same infrastructure and issues
as UbiComp and its personalization.
AR = enhance personal awareness (as needed) of: physical & nonphysical context, personal state, opportunities & threats...local & distant.
AR = deliver personalized info with less effort, more space & time
precision, and fewer seams than before, without increasing distraction.
Scope - General
AR = subset of Ubiquitous Computing: same infrastructure and issues
as UbiComp and its personalization.
AR = enhance personal awareness (as needed) of: physical & nonphysical context, personal state, opportunities & threats...local & distant.
AR = deliver personalized info with less effort, more space & time
precision, and fewer seams than before, without increasing distraction.
Augmented Context + Information Salience = Value
Scope - Model
1st plateau of AR = augmented perception of internal, external, &
distant reality. Smartphone and headset AR that’s non-intrusive.
Beyond that, the value of well-evolved AR has significant parallels to
the value of consciousness:
2nd plateau of AR = augmented decision-making. Integrated analysis,
evaluation, and management (informatics) of data streams, constantly
and dynamically evaluating the salience of each stream in context of
the user’s ever-changing current situation and the user’s historical
paths through n dimensions of data.
Then displaying the most salient data...when appropriate: Contextual
evaluation driving contextual augmentation; software ‘awareness’
augmenting the decision process via delivery of filtered salient aspects
of the user’s situation. At least 2 generations further.
Scope - Model
The value of well-evolved AR has significant parallels to the value of
consciousness, AKA:
...the Default Mode Network of the brain
•
One of the hardest-working systems in the brain
•
Active when the brain is aware, but not focused on a particular task
•
Reduces activity when you concentrate on a specific task
•
2 linked brain areas:
A) Imagination, empathy for the thoughts and intent of others
B) Personal memories, visualization of future actions, scenarios
Scope - Model
The value of well-evolved AR has significant parallels to the value of
consciousness, AKA:
...the Default Mode Network of the brain
•
One of the hardest-working systems in the brain
•
Active when the brain is aware, but not focused on a particular task
•
Reduces activity when you concentrate on a specific task
•
2 linked brain areas:
A) Imagination, empathy for the thoughts and intent of others
B) Personal memories, visualization of future actions, scenarios
DFM = constant process of considering how to be & what to do next.
Getting and dealing with the data upon which to make all those
decisions is the constant background task of 2nd-plateau AR apps.
Scope - Phone
AR hot again as part of the smartphone ecosystem.
Smartphone AR (SAR?) = low-hanging fruit. 2nd is games.
SAR & LBS* = features, not an industry. Startups will fail/be acquired.
Privacy / security issues will be more extreme than currently.
Acronym Fusion: AR + LBS + GIS + Gesture = POTS... “my computer”
*LBS =
Scope - farther
AR hot again as part of the smartphone ecosystem.
Smartphone AR (SAR?) is the low-hanging fruit.
SAR & LBS = features, not an industry; startups will fail / be acquired.
Privacy / security issues will be more extreme.
Acronym Fusion: AR + LBS + GIS = POTS... “my computer”
After that:
Contextual Personalization + More Sensors + onboard DBs + Projectors
Background
My work
Product strategy, dev., design, improvement, & IP evaluation, all based
on User-Centered Design.
For emerging technologies and markets in software, consumer
electronics, and entertainment.
My perspective
“Product as Tool”
Tool should adapt to (fit) the user and be its own user manual.
Scythes are an excellent model system for tool design, though we
don’t have ~2000 years to iterate.
Background
AR-related projects:
VideoCart, ‘92: Location-aware mobile retail touch-screen kiosk system
with data-driven path mapping.
Live Wall, ‘98: 100-meter, rear-projection, 1:1 scale videophone.
“Ambient Intelligence” strategy,’98 for Philips Consumer Electronics.
Royal Tyrell Museum, Alberta, ‘06: Concept & spec: permanent LBSbased virtual paleontological exhibit complex across 11 remote
archaeological sites, for Reich + Petch.
Dubailand theme parks, ‘08: Concept evaluation & high-level system
spec, strategic & tech partners: 24/7 multi-application smartphone AR
experience.
VideoCart: mobile location-aware touchscreen retail, ‘92
VideOcart
Live Wall: semi-mobile videophone design for Philips, ‘98
“Ambient Intelligence” strategy for Philips, ‘98
all photos: Philips
Dubailand AR project, ‘08
• Concept > smartphone AR
• Technology identification, assessment, & qualification
• High-level system definition
• Strategic partnering
• Broadened scope to include services across all hotel, leisure, retail,
restaurant, and transit in Dubailand, and to
follow the visitors home.
Part of
Dubailand model:
c. Lynne LaCascia, all rights reserved
Dubailand AR project, ‘08
c. Lynne LaCascia, all rights reserved
All images © Chimera Design 2007-2008
AR Timeline
Rear-view
mirrors
Sailboat
tell-tales
Commercial
aviation use
1st jet fighter
heads-up
display
1958
Prehistory, ~1900
soon after Horse
invention carriages
of the sail.
Dedicated
tools begin:
1970s
ARToolkit
UW HITLab
1st car HUD
Olds Cutlass
Helmet- Supreme “Augmented
Useful smartphone AR
environments,”
sensor nets
mounted
displays
1974-8
Battery
energydensity
resolved
1988
1992+
1999
US DoD
“UbiComp”: Tom
air-combat Mark Weiser Caudell &
David Mizell
testing
Nokia’s
MARA
2006
Now
20??
Hands-up displays
Reed sails on boats in the prehistoric
Middle East had inherent sail-edge
tell-tales.
 Example of sensor/display fusion
photo c. Luxury Peru Tours & Travel
Marmon Wasp racecar, 1911
Non-see-through
HUD display:
Rear-view mirror
is just above line
of sight
HUDs
Cintel “Buccaneer”
aircraft cockpit heads-up
display device, UK 1958
photo: Rochester Avionics Archives, UK
MIG21 heads-up display
(installed): circa 1960s
photo: Will Hise cc
HUD: modern commercial aviation AR
HMDs
Helmet-mounted displays
• High-priority military R&D investment
• Tethered to power and computer
• Obstructs part of the field of view
• Delicate to align, adjust, maintain
c. Michael Edwards,2007
• Size and weight unacceptable
• Adds complexity/distraction to user situation
• Current state of the art =
net loss of context awareness
Rockwell-Collins Sim Eye SR100A, 2009
HMDs
Helmet-mounted displays
• High-priority military R&D investment
• Tethered to power and computer
• Obstructs part of the field of view
• Delicate to align, adjust, maintain
c. Michael Edwards,2007
• Size and weight unacceptable
• Adds complexity/distraction to user situation
• Current state of the art =
net loss of context awareness

Not AR
Rockwell-Collins Sim Eye SR100A, 2009
Hands-Up Displays
Mobilizy's original product, “Wikitude AR (Travel) Guide,” for Android
photo: Mobilizy GmbH
Some Players
Mobilizy
• Wikitude World Browser,
• new Wikitude Drive (Android)
Nokia
• Point & Find, Image Space, MARA
Total Immersion
• D’Fusion Studio, Pro, Mobile
Metaio
• Unify Design: no programming
Tonchidot
Layar
• Layar Reality Browser 2.1 (Android)
• Enables adding layers easily
• Example: Trulia.com
• Working on iPhone dev now
• Sekai Camera (iPhone & Android)
Augmento
....
Studierstube ES
• Rich development framework
• Client and middleware
• 3D object tracking
photo: Mobilizy GmbH
Copyright Network World, Inc. All rights reserved.
Studierstube ES
photo: Mobilizy GmbH
Copyright Network World, Inc. All rights reserved.
Hardware
Now
Need
•
•
•
•
•
•
•
• Near-field RFID (“touch,” “wand”)
GPS
Accelerometers
Light sensors
Proximity sensors
Compass
Touch (2- or 2.5-D)
Bluetooth
• Constant visual awareness + recording
• Constant audio awareness + recording
• Galileo-grade location resolution
• Ultrasonic & IR spatial analysis
• Laser projector(s) with geometric
awareness & beam stabilization for
object augmentation
• ...
Software
Next--1.
Constant, robust markerless 2D & 3D feature
I.D. and object recognition (Nokia Point &
Find, ...) as a general feature
2.
Motion analysis + evaluation linked to the
object recognition
3.
General-purpose indoor position sensing
4.
More nuanced gestural input interpretation +
integration with voice
5.
Constant audio + visual awareness with
Software
Still further out...
Rob Cook’s award talk at Siggraph ‘09:
“...the next holy grail for the computer graphics community should be
realist augmented reality.”
http://media.siggraph.org/s2009podcasts/RobCook_AwardExport.mov
Augmenting Revenue
Assumptions
• Full OS citizenship: no tethering to a computer
• More coverage & bandwidth
• CPU & memory follow Moore’s Law
• Something magic happens to battery energy density.
Work
• Extend existing software onto the mobile platform with AR where
appropriate. Includes Augmenting Security for public locations.
• Building AR into new apps as a normal part of software development: Like the shift to web from desktop & enterprise.
• Multi-modal interface differentiation:
Handheld vs. plugged into a desktop screen or large projector
Platforms and toolkits
Platforms
iPhone 3.9
Android 1.5
Palm WebOS
RIM BlackBerry OS
Symbian S60 5th Edition
Windows Mobile 6.5
(Samsung LinuxOS?)
Tools
Studierstube ES
Layar
ARToolkit
NYARToolkit
FLARToolkit
Tinmith
...
[email protected]
end OA logo
AR-related conferences
Ubicomp 2009, Orlando, FL. Sept. 30-Oct. 3, 2009 www.ubicomp.org/ubicomp2009/
Int’l Symposium on Mixed and Augmented Reality (ISMAR09) - Orlando, FL, October 19-22, 2009 http://ismar09.org
8th Int’l Conference on Mobile and Ubiquitous Multimedia (MUM 2009) 22-25 Nov., Cambridge, UK www.mum2009.org
8th IEEE Int’l Conf. on Pervasive Computing & Communications (PerCom) Mannheim (Mar 29-April 2, 2010)
www.percom.org
7th IEEE Workshop on Ubiquitous Communications & Services: www.ubiquitous-management.org/mucs/2010/index.phpInt
Symposium on Wearable Computers (ISWC'09) - Linz, Austria, September 4-7, 2009. www.iswc.net/register
IEEE VR March 20-26, 2010 in Waltham, Massachusetts http://conferences.computer.org/vr/2010/
IEEE Symposium on 3D User Interfaces (3DUI), March 20-21 2010, Waltham, Mass
http://conferences.computer.org/3dui/3dui2010/
Symposium on Haptic Interfaces March 25-26 2010, Waltham, Mass., www.hapticssymposium.org/next_conference.htm
Int’l Conference on Mobile, Wireless and Optical Communication (MWOC 2009), Dec. 09 Korea
www.iacsit.org/mwoc/index.htm
Interservice/Industry Training, Simulation and Education Conference (I/ITSEC) Nov. 30-Dec. 3 Orlando, FL www.iitsec.org/
SVC Wireless Annual Conference Sep 25-26, 2009, Mountain View www.svcwireless.org/svcw/ac2009-home
Location Intelligence Conference 2009 Oct 5-7 2009, Broomfield, CO www.locationintelligence.net/
3rd Int’l Conference on Mobile Ubiquitous Computing, Systems, Services & Technologies UBICOMM October 11-16, 2009 in Sliema, Malta www.iaria.org/conferences2009/UBICOMM09.html
Mobiquitous 2010 www.mobiquitous.org/
2nd ACM Int’l Workshop on Mobile Opportunistic Networking Feb. 22-23, 2010, Pisa, http://cnd.iit.cnr.it/mobiopp2010
2nd Int’l Conference on Mobile Lightweight Wireless Systems (MOBILIGHT 2010) www.mobilight.org
5th Int’l Conference on Body Area Networks September (BodyNets) Sept. 10-12, 2010 Corfu, Greece www.bodynets.org/
IMMERSECOM 2010 http://immerscom.org/ (2009 was at Berkeley)
1st Int’l ICST Conference on User Centric Media (UCMedia 2009), 9-11 Dec. 09, Venice www.usercentricmedia.org
Credits
•
•
•
•
•
•
•
•
•
•
•
•
Default Mode Network paper: Gusnard, DA, Raichle, ME: “Searching for a baseline: the
resting human brain from the perspective of functional imaging.” Nature Reviews
Neuroscience, 2001
“iLamps: Geometrically Aware and Self-Configuring Projectors,” Rascar, et al. 2003
Commercial aviation HUD image courtesy Todd Lapin, cc
MARA image courtest Nokia Research
Handheld AR.net (http://handheldar.net/stbes.php) for Studierstube ES diagram
Fighter jet HUD device image cc www.rochesteravionicarchives.co.uk
Photo of sail’s tell-tales courtesy Bill Gracey, cc
Modern helmet-mounted see-through display courtesy Rockwell-Collins: "SIM EYE SR100-A”
Photo of Ray Harroun’s Marmon Wasp racecar with mirror, cc The359
Vuzix Wrap 920AV and Tac-Eye GMD images c. Vuzix Inc.
Theme park illustrations © Chimera Design, 2007-2009
Photo of Dubailand model © Lynne LaCascia, 2009
Software Development Forum
Emerging Technology SIG
Augmented Reality
Maribeth Back, Fuji-Xerox Palo Alto Labs
“High Tech Chocolate:
Exploring Mixed Reality applications for Industry”
[email protected]