Transcript The Vision and Reality of Ubiquitous Computing Prof. Henning Schulzrinne Dept. of Computer Science Columbia University (with Arezu Moghadam, Ron Shacham, Suman Srinivasan, Xiaotao Wu and.

The Vision and Reality of Ubiquitous
Computing
Prof. Henning Schulzrinne
Dept. of Computer Science
Columbia University
(with Arezu Moghadam, Ron Shacham, Suman Srinivasan,
Xiaotao Wu and other IRT members; parts in cooperation
with DoCoMo Eurolabs)
August 8, 2007
Mobiquitous 2007
Overview
• The original vision of ubiquitous
computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality: 7DS
August 8, 2007
Mobiquitous 2007
Ubiquitous computing  ubiquitous
communications
• “It is invisible, everywhere computing that does not live on a
personal device of any sort, but is in the woodwork everywhere.”
• Weiser’s original vision (“Nomadic Issues in Ubiquitous Computing”,
1996)
–
–
–
–
–
“one person, many computers”
many computers embedded in environment
dynamic ownership
PC phonebooth
“IR use will grow rapidly”
• Updated version, 2007
–
–
–
–
–
not physically invisible, but transparent
emphasis on communications, not computing
most devices are mobile (or nomadic)
cheap electronics personal devices
radio (channelized and UWB)
August 8, 2007
Mobiquitous 2007
Overview
• The original vision of ubiquitous
computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality: 7DS
August 8, 2007
Mobiquitous 2007
User challenges vs. research challenges
• Are we addressing real user needs?
– Engineering vs. sports
• My guesses
ease of use
reliability
no manual
no re-entry
no duplication
integration
phishing
data loss
cost
August 8, 2007
limited risk
Mobiquitous 2007
Example: Email configuration
• Application configuration
for (mobile) devices painful
• SMTP port 25 vs. 587
• IMAP vs. POP
• TLS vs. SSL vs. “secure
authentication”
• Worse for SIP...
August 8, 2007
Mobiquitous 2007
Example: SIP configuration
•
•
•
•
•
partially explains
highly technical parameters, with differing names
inconsistent conventions for user and realm
made worse by limited end systems (configure by multi-tap)
usually fails with some cryptic error message and no
indication which parameter
out-of-box experience not good
August 8, 2007
Mobiquitous 2007
Mobile why’s
•
•
•
•
•
•
•
•
•
Not research, but examples of real annoyances
Why does each mobile device need its own power supply?
Why do I have to adjust the clock on my camera each time I travel?
Why do I have to know what my IMAP server is and whether it uses
TLS or SSL?
Why do I have to type in my address book?
Why do I have to “synchronize” my PDA?
Why do I have to manually update software?
Why is connecting a laptop to a projector a gamble?
Why do we use USB memory sticks when all laptops have 802.11b?
August 8, 2007
Mobiquitous 2007
Consumer wireless & mobile devices
Prius key
Garage door opener
TAN display
Water leak alarm
wireless door bell
MSN Direct weather
August 8, 2007
Mobiquitous 2007
Mobile systems - reality
GPS
•
idea: special purpose (phone) --> universal
communicator
–
•
idea is easy...
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
mobile equipment: laptop + phone
–
•
•
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
sufficiently different UI and capabilities
location data
we all know the ideal (converged) cell phone
difficulty is not technology, but integration
and programmability
–
–
–
–
(almost) each phone has a different flavor of
OS
doesn’t implement all functionality in Java
APIs
no dominant vendor (see UNIX/Linux vs.
Microsoft)
external interfaces crippled or unavailable
•
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
e.g., phone book access
QuickTime™ and a
TIFF (Uncompressed) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF ( Uncompressed) decompr essor
are needed to see this pictur e.
August 8, 2007
Mobiquitous 2007
Example: displays and speakers
August 8, 2007
Mobiquitous 2007
The mobile ubiquitous challenge
Mobile phone
Mobile Internet access
Interconnected devices
“Internet
of things”
August 8, 2007
Mobiquitous 2007
What do we need?
• Standards, not new technology
• Radio connectivity
– 802.11a/b/g/n, 802.15.4 
– better discovery of networks
• Location information everywhere
• Discovery: devices & services
– network-local discovery via Bonjour (mDNS) 
– missing: location-based discovery
• Advanced mobility: session, personal, service
• Event notification
• Data formats
– location, sensor events, ...
August 8, 2007
Mobiquitous 2007
Examples of “invisible” behavior
• MP3 player in car automatically picks up new files in home server
• A new email with vcard attachment automatically updates my cell
phone address book
• The display of my laptop appears on the local projector
– without cable or configuration
• I can call people I just met at Mobiquitous
– without exchanging business cards
•
•
•
•
My car key opens my front door
My cell phone serves as a TAN (one-time password) generator
My cell phone automatically turns itself off during a lecture
My camera knows where the picture was taken
August 8, 2007
Mobiquitous 2007
An interconnected system
opens doors
generates TAN
incoming call
updates location
time, location
address book
alert, events
any weather service
school closings
August 8, 2007
acoustic alerts
Mobiquitous 2007
Thinking beyond 802.11 and UMTS
• Many interesting networks beyond those covered in conferences
– ease of access by researchers vs. importance
– 90% of papers on 802.11b and maybe GPRS, BlueTooth
• New wireless networks
–
–
–
–
–
–
–
–
broadcast instead of unicast -- useful for many ubiquitous applications
S5 for low-rate sensors (city scale)
QuickTime™ and a
Zigbee (802.15.4) for local sensors (20 - 250 kb/s)
TIFF (Uncompressed) decompressor
are needed to see this picture.
FM subcarrier (not really new) -- MSN Direct
FM Radio Data System -- TMC
Sirius / XM
HD radio
paging
August 8, 2007
Mobiquitous 2007
Overview
• The original vision of ubiquitous
computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality: 7DS
August 8, 2007
Mobiquitous 2007
Application-layer mobility
• terminal mobility
– one terminal, multiple network addresses
• Personal mobility
– one person, multiple terminals
– e.g., Grandcentral
• session mobility
– one user, multiple terminals in sequence or in parallel
• service mobility
– services move with user
August 8, 2007
Mobiquitous 2007
Session mobility
•
Walk into office, switch from cell
phone to desk phone
– call transfer problem  SIP
REFER
•
related problem: split session
across end devices
– e.g., wall display + desk phone +
PC for collaborative application
– assume devices (or stand-ins) are
SIP-enabled
– third-party call control
August 8, 2007
Mobiquitous 2007
R. Shacham, H. Schulzrinne, S. Thakolsri, W. Kellerer,
“Ubiquitous device personalization and use: The next generation
of IP multimedia communications”, ACM TOMCCAP, May 2007
How to find services?
•
Two complementary developments:
–
–
smaller devices carried on user instead of stationary
devices
devices that can be time-shared
•
•
•
•
•
•
Need to discover services in local environment
–
SLP (Service Location Protocol) allows querying for
services
•
•
–
–
–
•
“find all color displays with at least XGA resolution”
slp://example.com/SrvRqst?public?type=printer
SLP in multicast mode
SLP in DA mode
Apple Bonjour
Need to discover services before getting to
environment
–
–
–
August 8, 2007
large plasma displays
projector
hi-res cameras
echo-canceling speaker systems
wide-area network access
“is there a camera in the meeting room?”
SLP extension: find remote DA via DNS SRV
LoST to find services by geographic location
Mobiquitous 2007
Session mobility
Local Devices
Transcoder
Internet
SLP DA
SLP SA
SLP UA
SIP SM
SIP UA
SIP UA
Correspondent
Node (CN)
August 8, 2007
SIP SM
SIP UA
SLP UA
Mobiquitous 2007
Mobile Node (MN)
SLP
SIP
RTP
Overview
• The original vision of ubiquitous
computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality: 7DS
August 8, 2007
Mobiquitous 2007
Context-aware communication
•
•
•
context = “the interrelated conditions in which something
exists or occurs”
anything known about the participants in the (potential)
communication relationship
both at caller and callee
time
chronology, uniqueness
capabilities
caller preferences
location
location-based call routing
location events (emergency alerts)
security (access control)
service discovery
activity/availability
presence
sensor data (mood, bio)
privacy issues similar to location data
August 8, 2007
Mobiquitous 2007
GEOPRIV and SIMPLE architectures
rule
maker
DHCP
XCAP
(rules)
target
presentity
caller
August 8, 2007
publication
interface
PUBLISH
INVITE
location
server
presence
agent
notification
interface
location
recipient
GEOPRIV
watcher
SIP
presence
SUBSCRIBE
NOTIFY
INVITE
Mobiquitous 2007
callee
SIP
call
Presence data architecture
presence sources
PUBLISH
create
view
(compose)
raw
presence
document
XCAP
select best source
resolve contradictions
depends on watcher
XCAP
privacy
policy
composition
policy
(not defined yet)
August 8, 2007
privacy
filtering
draft-ietf-simple-presence-data-model
Mobiquitous 2007
Presence data architecture
candidate
presence
document
remove data not of
interest
watcher
August 8, 2007
raw
presence
document
watcher
filter
post-processing
composition
(merging)
SUBSCRIBE
difference
to previous notification
NOTIFY
Mobiquitous 2007
final
presence
document
Presence data model
person
“calendar”
“cell”
“manual”
(presentity)
(views)
services
[email protected]
audio, video, text
[email protected]
video
devices
August 8, 2007
Mobiquitous 2007
RPID = rich presence
•
•
Provide watchers with better information about the what, where, how of
presentities
facilitate appropriate communications:
–
–
–
•
designed to be derivable from calendar information
–
•
“wait until end of meeting”
“use text messaging instead of phone call”
“make quick call before flight takes off”
or provided by sensors in the environment
allow filtering by “sphere” – the parts of our life
–
don’t show recreation details to colleagues
August 8, 2007
Mobiquitous 2007
Rich presence
• More information: for (authorized) people and applications
• automatically derived from
– sensors: physical presence, movement
– electronic activity: calendars
• Rich information:
–
multiple contacts per presentity
•
•
–
–
–
–
–
–
device (cell, PDA, phone, …)
service (“audio”)
activities, current and planned
sphere (home vs. work)
current user mood
surroundings (noise, privacy, vehicle, …)
contact information
composing (typing, recording audio/video IM, …)
August 8, 2007
Mobiquitous 2007
Presence and privacy
•
•
All presence data, particularly
location, is highly sensitive
Basic location object (PIDFLO) describes
– distribution (binary)
– retention duration
•
Policy rules for more detailed
access control
– who can subscribe to my
presence
– who can see what when
August 8, 2007
<tuple id="sg89ae">
<status>
<gp:geopriv>
<gp:location-info>
<gml:location>
<gml:Point gml:id="point1“
srsName="epsg:4326">
<gml:coordinates>37:46:30N 122:25:10W
</gml:coordinates>
</gml:Point>
</gml:location>
</gp:location-info>
<gp:usage-rules>
<gp:retransmission-allowed>no
</gp:retransmission-allowed>
<gp:retention-expiry>2003-06-23T04:57:29Z
</gp:retention-expiry>
</gp:usage-rules>
</gp:geopriv>
</status>
<timestamp>2003-06-22T20:57:29Z</timestamp>
</tuple>
Mobiquitous 2007
Events as missing Internet capability
• aka PUB/SUB
• Used across applications, e.g.,
–
–
–
–
–
–
–
–
email and voicemail notification
presence
replace RSS (= poll!)
web service completion
emergency alerts (“reverse 9-1-1”)
network management
home control
data synchronization
• Rich research history
– but too complex, optimize the wrong thing
• XMPP and SIP as likely transport candidates
August 8, 2007
Mobiquitous 2007
Local Switch
Automatic
Number
Identification
Automatic
Location
Identification
August 8, 2007
Collaboration between
local phone providers and
local public safety agencies
Mobiquitous 2007
911 technology failures
•
NY Times (“An S O S for 911 Systems in Age of High-Tech”), 4/6/07:
– “40% of … counties, most of them rural or small-town …, cannot yet pinpoint the
location of the cellphone callers, though the technology to do so has been
available for at least five years.”
“In … Okmulgee, Okla., last November, 4-year-old Graciella Mathews-Tiger died
in a house fire after a 911 operator who lacked the technology to pinpoint the call
misheard the address.”
• Phase II wireless; billions of dollars spent
• In Mississippi, only 1 of out 5 counties
– “As it ages, it is cracking, with problems like system overload, understaffing,
misrouted calls and bug-ridden databases leading to unanswered calls and
dangerous errors.”
• operator (CAMA) trunks, with 8-digit number delivery
• MSAG and ALI databases
August 8, 2007
Mobiquitous 2007
Location delivery
GPS
HELD
HTTP
wire
map
DHCP
LLDP-MED
August 8, 2007
Mobiquitous 2007
Location, location, location, ...
Voice Service Provider (VSP)
sees emergency call
but does not know caller location
ISP/IAP knows user location
but does not handle call
August 8, 2007
Mobiquitous 2007
Options for location delivery
•
Wireless
–
–
–
–
•
GPS
S5 wireless (active sensors + triangulation)
Skyhook (802.11 in urban areas)
HDTV
L2: LLDP-MED (standardized version of CDP + location data)
– periodic per-port broadcast of configuration information
•
L3: DHCP for
– geospatial (RFC 3825)
– civic (RFC 4676)
•
L7: proposals for retrievals: HELD, SIP, …
–
–
–
–
–
for own IP address or by third party (e.g., ISP to infrastructure provider)
by IP address
by MAC address
by identifier (conveyed by DHCP or PPP)
HTTP-based
August 8, 2007
Mobiquitous 2007
Locating Caller using LLDP-MED
LLDP-MED stands for: *
* From Wikipedia
Link Layer Discovery Protocol
“a vendor-neutral Layer 2 protocol that allows a network device
to advertise its identity and capabilities on the local network.”
Media Endpoint Discovery
“an enhancement to the LLDP that allows discovery of other
things including location “
“I am LLDP-MED Capable.
I can process location information.”
LLDP-MED
SWITCH
CALLER
EQUIPMENT
August 8, 2007
“Your location is:
500 W 120TH st. New York NY 10027”
Mobiquitous 2007
Location determination options
Method
CDP or LLDPMED
DHCP
HELD
GPS
manual entry
Layer
L2
L3
L7 (HTTP)
-
user
advantages
• simple to
implement
• built into switch
• direct port/room
mapping
• simple to
implement
• network
locality
• traverses
NATs
• can be
operated by
L2 provider
• accurate
• mobile
devices
• no carrier
cooperation
• no
infrastructure
changes
• no carrier
cooperation
problems
may be hard to
automate for large
enterprises
mapping MAC
address to
location?
mapping IP
address to
switch port?
• indoor
coverage
• acquisition
time
• fails for
mobile
devices
• unreliable for
nomadic
Use
Ethernet LANs
Enterprise
LANs
Some ISPs
DSL, cable
mobile devices
fall back
August 8, 2007
Mobiquitous 2007
SIP message for Location Info.
INVITE urn:service:sos SIP/2.0
To: urn:service:sos
Call-ID: [email protected]
Via: SIP/2.0/TCP 192.168.1.106:4064;rport
Content-Type: multipart/mixed; boundary
From: sip:[email protected]
Contact: <sip:[email protected]:5060>
CSeq: 1 INVITE
Content-Length: 1379
request line
header fields
------ =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY=
MIME-Version: 1.0
content-Type: application/sdp
Content-Transfer-Encoding: 8bit
v=0
o=eddie 1127764654 1127764654 IN IP4 192.168.1.106
s=SIPC Call
c=IN IP4 160.39.54.70
t=0 0
m=audio 10000 RTP/AVP 0 3
m=video 20000 RTP 31
SDP
August 8, 2007
------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY=
MIME-Version: 1.0
Content-Type: application/pidf+xml
Content-Transfer-Encoding: 8bit
<?xml version="1.0" encoding="ISO-8859-1"?>
<presence xmlns="urn:ietf:params:xml:ns:pidf"
xmlns:gp="urn:ietf:params:xml:ns:pidf:geopriv10"
xmlns:cl=" urn:ietf:params:xml:ns:pidf:geopriv10:civilLoc"
xmlns:gml="urn:opengis:specification:gml:schema-xsd:feature:v3.0"
entity="sip:[email protected]">
<tuple id="28185">
<status>
<gp:geopriv>
<gp:location-info>
<cl:civilAddress>
<cl:country>us</cl:country>
<cl:A1>ny</cl:A1>
<cl:A2>new york</cl:A2>
<cl:A3>new york</cl:A3>
<cl:A6>amsterdam</cl:A6>
<cl:HNO>1214</cl:HNO>
</cl:civilAddress>
</gp:location-info>
<gp:method>Manual</gp:method>
</gp:geopriv>
</status>
<contact priority="0.8">sip:[email protected]:5060</contact>
<timestamp>2005-09-26T15:57:34-04:00</timestamp>
</tuple>
</presence>
------- =_ZGY1NTFlZDJkMDkxY2FkMTIxMWI2MzIzNjE1M2U0OTY=--
Mobiquitous 2007
PIDF-LO
August 8, 2007
Mobiquitous 2007
Tracking
August 8, 2007
Mobiquitous 2007
Data formats: location
• Civic (street)
– jurisdictional & postal
• Geo (longitude & latitude)
– point, polygon, circle, …
• see GeoRSS for simple example
August 8, 2007
Mobiquitous 2007
ECRIT: LoST Functionality
•
Civic as well as geospatial queries
–
•
•
civic address validation
Recursive and iterative resolution
Fully distributed and hierarchical
deployment
–
–
•
Indicates errors in civic location data 
debugging
–
•
but provides best-effort resolution
Can be used for non-emergency services:
–
–
directory and information services
pizza delivery services, towing companies, …
can be split by any geographic or civic
boundary
same civic region can span multiple
LoST servers
August 8, 2007
<findService xmlns="urn:…:lost1">
<location profile="basic-civic">
<civicAddress>
<country>Germany</country>
<A1>Bavaria</A1>
<A3>Munich</A3>
<A6>Neu Perlach</A6>
<HNO>96</HNO>
</civicAddress>
</location>
<service>urn:service:sos.police</service>
</findService>
Mobiquitous 2007
LoST: Location-to-URL Mapping
VSP1
cluster serving VSP1
replicate
root information
cluster
serves VSP2
123 Broad Ave
Leonia
Bergen County
NJ US
LoST
NJ
US
root
NY
US
sip:[email protected]
nodes
search
referral
Bergen County
NJ US
Leonia
NJ US
August 8, 2007
Mobiquitous 2007
LoST Architecture
G
tree guide
G
G
G
T1: .us
G
broadcast (gossip)
T2: .de
resolver
seeker
313 Westview
Leonia, NJ US
T2
T1
(.us)
August 8, 2007
(.de)
T3
(.dk)
Leonia, NJ  sip:[email protected]
Mobiquitous 2007
Left to do: event notification
•
notify (small) group of users when something of interest happens
–
–
–
–
–
•
presence = change of communications state
email, voicemail alerts
environmental conditions
vehicle status
emergency alerts
kludges
– HTTP with pending response
– inverse HTTP --> doesn’t work with NATs
•
•
Lots of research (e.g., SIENA)
IETF efforts starting
– SIP-based
– XMPP
August 8, 2007
Mobiquitous 2007
Overview
• The original vision of ubiquitous
computing
• User challenges
• Beyond terminal mobility
• Location as new core service
• Universality: 7DS
August 8, 2007
Mobiquitous 2007
Problems with Wide Area Wireless
• 802.11 currently hard to deploy across city or large area
• Problem: How can mobile devices / gadgets get information
while on the move?
• Use local peer-to-peer wireless networks to exchange
information
– Peers can get information they do not have from another peer
Solution: 7DS!
August 8, 2007
Mobiquitous 2007
S. Srinivasan, A. Moghadam, S.G. Hong, H. Schulzrinne,
“7DS - Node Cooperation and Information Exchange in
Mostly Disconnected Networks”, ICC '07. June 2007.
How 7DS Works
1. When devices are in the same BSS (Basic service set) of
802.11 ad-hoc network, they discover each other using
service discovery of Zeroconf
zeroconf
August 8, 2007
Mobiquitous 2007
How 7DS Works
2. If there is no Internet connection, the devices can communicate
with each other to exchange information
August 8, 2007
Internet
Mobiquitous 2007
Web Delivery Model
• 7DS core functionality: Emulation of web content access
and e-mail delivery
August 8, 2007
Mobiquitous 2007
Design
• Peer-to-peer network set up using zeroconf
– Protocol enables devices to communicate with each other
without a DHCP server, a DNS server and a Directory server
•
•
•
•
August 8, 2007
Proxy server serves content
Search engine searches for local data
MTA store and forward
In progress: File synchronization, BBS
Mobiquitous 2007
Store and Forward
• Forwarding e-mail in the ad-hoc network
• Acts as an MTA
August 8, 2007
Mobiquitous 2007
Search Engine
• Provides ability to query self
for results
• Searches the cache index
using Swish-e library
• Presents results in any of three
formats: HTML, XML and plain
text
• Similar in concept to Google
Desktop
August 8, 2007
Mobiquitous 2007
Query Multicast Engine
•
•
•
•
Used to actually exchange
information among peers
Requesting peer broadcasts a
query to the network
Responding peers reply if they
have information
– Send encoded string with list
of matching items
Requesting peer retrieves suitable
information
August 8, 2007
Mobiquitous 2007
File Synchronization
SERVICE
ANNOUNCEMENT
FILE SYNCHRONIZATION
RESOLUTION USING RSYNC PROTOCOL
SRV : 7ds-fs1.filesync._7ds._udp.local.
“I want
7ds-device1.local:2525
Word.doc and presentation.ppt”
TXT : file1.xml
TXT : file2.xml
SRV : 7ds-fs2.filesync._7ds._udp.local.
“I want
7ds-device2.local:2525
File1.xml
and file2.xml”
TXT : word.doc
File1.xml
File2.xml
TXT : presentation.ppt
Word.doc
Presentation.ppt
August 8, 2007
Mobiquitous 2007
Word.doc
Presentation.ppt
File1.xml
File2.xml
Conclusion
• Motivate mobile ubiquitous research by user problems
• From stovepipe mobile & wireless systems to personal
and shareable wireless networks
• Thinking beyond single applications
– presence event notification
– 9-1-1 location  time & location as infrastructure
• Need new models of creating services
– domain-specific languages, not Java APIs
August 8, 2007
Mobiquitous 2007