Transcript Document

Implementation and Research
Issues in Query Processing
for Wireless Sensor Networks
Wei Hong
Intel Research, Berkeley
[email protected]
Sam Madden
MIT
[email protected]
1
MDM Tutorial, January 19th 2004
Motivation
• Sensor networks (aka sensor webs, emnets) are here
– Several widely deployed HW/SW platforms
• Low power radio, small processor, RAM/Flash
– Variety of (novel) applications: scientific, industrial, commercial
– Great platform for mobile + ubicomp experimentation
• Real, hard research problems to be solved
– Networking, systems, languages, databases
• We will summarize:
– The state of the art
– Our experiences building TinyDB
– Current and future research directions
Berkeley
Mote
2
Sensor Network Apps
Habitat Monitoring: Storm
petrels on Great Duck Island,
microclimates on James
Reserve.
Vehicle detection: sensors along a
road, collect data about passing
vehicles.
Earthquake monitoring in shaketest sites.
Just the tip of the iceberg
-- more tomorrow!
Traditional monitoring
3
apparatus.
Declarative Queries
• Programming Apps is Hard
–
–
–
–
–
Limited power budget
Lossy, low bandwidth communication
Require long-lived, zero admin deployments
Distributed Algorithms
Limited tools, debugging interfaces
• Queries abstract away much of the complexity
– Burden on the database developers
– Users get:
• Safe, optimizable programs
• Freedom to think about apps instead of details
4
TinyDB: Prototype declarative
query processor
• Platform: Berkeley Motes + TinyOS
• Continuous variant of SQL : TinySQL
• Power and data-acquisition based innetwork optimization framework
• Extensible interface for aggregates, new
types of sensors
5
Agenda
• Part 1 : Sensor Networks (50 Minutes)
– TinyOS
– NesC
• Short Break
• Part 2: TinyDB (1 Hour)
– Data Model and Query Language
– Software Architecture
• Long Break + Hands On
• Part 3: Sensor Network Database Research
Directions (1 Hour, 10 Minutes)
6
Part 1
• Sensornet Background
• Motes + Mote Hardware
– TinyOS
– Programming Model + NesC
• TinyOS Architecture
– Major Software Subsystems
– Networking Services
7
A Brief History of Sensornets
• People have used sensors for a long time
• Recent CS History:
– (1998) Pottie + Kaiser: Radio based networks of
sensors
– (1998) Pister et al: Smart Dust
• UCLA / USC / Berkeley Continue to Lead Research
• Initial focus on optical communication
•Many other players now
• By 1999, radio based networks, COTS Dust, “Motes”
•TinyOS/Motes as most common platform
– (1999) Estrin + Govindan
• •Ad-hoc
networks
of sensors
Emerging
commercial
space:
– (2000)
Culler/Hill
et al:
TinyOS
+ Motes
• Crossbow,
Ember,
Dust,
Sensicast,
Moteiv, Intel
– (2002) Hill / Dust: SPEC, mm^3 scale computing
8
Why Now?
• Commoditization of radio hardware
– Cellular and cordless phones, wireless
communication
– (some radio pictures, etc.)
• Low cost -> many/tiny -> new applications!
• Real application for ad-hoc network
research from the late 90’s
• Coming together of EE + CS communities
9
Motes
Mica Mote
4Mhz, 8 bit Atmel RISC uProc
40 kbit Radio
4 K RAM, 128 K Program
Flash, 512 K Data Flash
AA battery pack
Mica2Dot
Based on TinyOS
10
History of Motes
• Initial research goal wasn’t hardware
– Has since become more of a priority with emerging
hardware needs, e.g.:
• Power consumption
• (Ultrasonic) ranging + localization
– MIT Cricket, NEST Project
• Connectivity with diverse sensors
– UCLA sensor board
– Even so, now on the 5th generation of devices
•
•
•
•
Costs down to ~$50/node (Moteiv, Dust)
Greatly improved radio quality
Multitude of interfaces: USB, Ethernet, CF, etc.
Variety of form factors, packages
11
Motes vs. Traditional
Computing
• Lossy, Adhoc Radio Communication
• Sensing Hardware
• Severe Power Constraints
12
Radio Communication
• Low Bandwidth Shared Radio Channel
– ~40kBits on motes
– Much less in practice
• Encoding, Contention for Media Access (MAC)
• Very lossy: 30% base loss rate
– Argues against TCP-like end-to-end retransmission
• And for link-layer retries
• Generally, not well behaved
13
From Ganesan, et al. “Complex Behavior at Scale.” UCLA/CSD-TR 02-0013
Types of Sensors
• Sensors attach via daughtercard
•Weather
–Temperature
–Light x 2 (high
intensity PAR, low
intensity, full
spectrum)
–Air Pressure
–Humidity
•Vibration
–2 or 3 axis
accelerometers
•Tracking
–Microphone (for ranging
and acoustic signatures)
–Magnetometer
• GPS
14
Power Consumption and
Lifetime
• Power typically supplied by a small battery
– 1000-2000 mAH
– 1 mAH = 1 milliamp current for 1 hour
• Typically at optimum voltage, current drain rates
– Power = Watts (W) = Amps (A) * Volts (V)
– Energy = Joules (J) = W * time
• Lifetime, power consumption varies by application
– Processor: 5mA active, 1 mA idle, 5 uA sleeping
– Radio: 5 mA listen, 10 mA xmit/receive, ~20mS / packet
– Sensors: 1 uA -> 100’s mA, 1 uS -> 1 S / sample
15
Energy Usage in A Typical
Data Collection Scenario
Percentage of Total Energy
Percentage of Total Power
• Each
mote collects 1 sample
of
Power Consumption Breakdown
Processor Energy Breakdown
(light,humidity) data every 10 seconds,
50
90
forwards it
45
80
• 70Each mote can “hear” 10 40
other motes
35
30
• 60Process:
50
25
20
15
10
5
0
– Wake up, collect samples (~ 1 second)
30– Listen to radio for messages to forward (~1
20 second)
10– Forward data
Idle
Wai ti ng Wai ti ng
40
0
Radio
Sensors
Processor
Hardware Element
for Radi o
Sendi ng
for
Sensors
Processing Phase
16
Sensors: Slow, Power Hungry,
Noisy
Time of
of Day
Day vs.
vs. Light
Light
Time
200
C hamber Sens or
C hamber Sens or
Sens or 6 9 (M edian of L as t 1 0 )
Sens or 6 9
180
160
140
Lux
Light
(Lux)
120
100
80
60
40
20
0
2 0 :0 9
2 0 :3 8
2 1 :0 7
2 1 :3 6
2 2 :0 4
2 2 :3 3
2
23
3:0
:02
2
-20
2
23
3:3
:31
1
00:0
:000
00:2
:288
00:5
:577
11:2
:266
17
Time
Time
of Day
Programming Sensornets:
TinyOS
• Component Based Programming Model
• Suite of software components
–
–
–
–
Timers, clocks, clock synchronization
Single and multi-hop networking
Power management
Non-volatile storage management
18
Programming Philosophy
• Component Based
– “Wiring” to components together via interfaces,
configurations
• Split-Phased
– Nothing blocks, ever.
– Instead, completion events are signaled.
• Highly Concurrent
– Single thread of “tasks”, posted and scheduled
FIFO
– Events “fired” asynchronously in response to
interrupts.
19
NesC
• C-like programming language with component model
support
– Compiles into GCC-compatible C
• 3 types of files:
– Interfaces
• Set of function prototypes; no implementations or variables
– Modules
• Provide (implement) zero or more interfaces
• Require zero or more interfaces
• May define module variables, scoped to functions in module
– Configurations
• Wire (connect) modules according to requires/provides
relationship
20
Component Example: Leds
module LedsC {
provides interface Leds;
}
implementation
{
uint8_t ledsOn;
enum {
RED_BIT = 1,
GREEN_BIT = 2,
YELLOW_BIT = 4
};
….
async command result_t Leds.redOn() {
dbg(DBG_LED, "LEDS: Red on.\n");
atomic {
TOSH_CLR_RED_LED_PIN();
ledsOn |= RED_BIT;
}
return SUCCESS;
}
….
}
21
Configuration Example
configuration CntToLedsAndRfm {
}
implementation {
components Main, Counter, IntToLeds, IntToRfm, TimerC;
}
Main.StdControl -> Counter.StdControl;
Main.StdControl -> IntToLeds.StdControl;
Main.StdControl -> IntToRfm.StdControl;
Main.StdControl -> TimerC.StdControl;
Counter.Timer -> TimerC.Timer[unique("Timer")];
IntToLeds <- Counter.IntOutput;
Counter.IntOutput -> IntToRfm;
22
Split Phase Example
module IntToRfmM { … }
implementation { …
command result_t IntOutput.output
(uint16_t value) {
IntMsg *message = (IntMsg *)data.data;
if (!pending) {
pending = TRUE;
message->val = value;
atomic {
message->src = TOS_LOCAL_ADDRESS;
}
if (call Send.send(TOS_BCAST_ADDR,
sizeof(IntMsg), &data))
return SUCCESS;
pending = FALSE;
}
return FAIL;
}
event result_t Send.sendDone
(TOS_MsgPtr msg,
result_t success) {
if (pending && msg == &data) {
pending = FALSE;
signal IntOutput.outputComplete
(success);
}
return SUCCESS;
}
}
}
23
Major Components
• Timers: Clock, TimerC, LogicalTime
• Networking: Send, GenericComm,
AMStandard, lib/Route
• Power Management:
HPLPowerManagement
• Storage Management: EEPROM, MatchBox
24
Timers
• Clock: Basic abstraction over hardware
timers; periodic events, single frequency.
• LogicalTime: Fire an event some number of
H:M:S:ms in the future.
• TimerC: Multiplex multiple periodic timers
on top of LogicalTime.
25
Radio Stack
• Interfaces:
– Send
• Broadcast, or to a specific ID
• split phase
– Receive
• asynchronous signal
• Implementations:
IntMsg *message = (IntMsg *)data.data;
…
message->val = value;
atomic {
message->src = TOS_LOCAL_ADDRESS;
}
call Send.send(TOS_BCAST_ADDR,
sizeof(IntMsg), &data))
– AMStandard
• Application specific messages
• Id-based dispatch
– GenericComm
• AMStandard + Serial IO
– Lib/Route
• Mulithop
event TOS_MsgPtr ReceiveIntMsg.
receive(TOS_MsgPtr m) {
IntMsg *message = (IntMsg *)m->data;
call IntOutput.output(message->val);
return m;
}
Wiring to equate IntMsg to ReceiveIntMsg
26
Multihop Networking
• Standard implementation “tree based
routing”
Problems:
A
Parent Selection
Asymmetric Links
Adaptation vs. Stability
Node D
Neigh Qual
B
.75
C
.66
E
.45
F
.82
Node C
Neigh Qual
A
.5
B
.44
D
.53
F
.35
B
B
R:{…}
B
B
C
B
R:{…}B
B
D
R:{…}B
R:{…}
B
R:{…}
B
B
B
F
E
B
27
Geographic Routing
• Any-to-any routing via geographic
coordinates
– See “GPSR”, MOBICOM 2000, Karp + Kung.
B
A
•Requires coordinate
system*
•Requires endpont
coordinates
•Hard to route
around local minima
(“holes”)
28
*Could be virtual, as in Rao et al “Geographic Routing Without Coordinate Information.” MOBICOM 2003
Power Management
• HPLPowerManagement
– TinyOS sleeps processor when possible
– Observes the radio, sensor, and timer state
• Application managed, for the most part
– App. must turn off subsystems when not in use
– Helper utility: ServiceScheduler
• Peridically calls the “start” and “stop” methods of an app
– More on power management in TinyDB later
– Approach works because:
• single application
• no interactivity requirements
29
Non-Volatile Storage
• EEPROM
– 512K off chip, 32K on chip
– Writes at disk speeds, reads at RAM speeds
– Interface : random access, read/write 256 byte
pages
– Maximum throughput ~10Kbytes / second
• MatchBox Filing System
– Provides a Unix-like file I/O interface
– Single, flat directory
– Only one file being read/written at a time
30
TinyOS: Getting Started
• The TinyOS home page:
– http://webs.cs.berkeley.edu/tinyos
– Start with the tutorials!
• The CVS repository
– http://sf.net/projects/tinyos
• The NesC Project Page
– http://sf.net/projects/nescc
• Crossbow motes (hardware):
– http://www.xbow.com
• Intel Imote
– www.intel.com/research/exploratory/motes.htm.
31
Part 2
The Design and Implementation
of TinyDB
32
Part 2 Outline
•
•
•
•
•
•
•
TinyDB Overview
Data Model and Query Language
TinyDB Java API and Scripting
Demo with TinyDB GUI
TinyDB Internals
Extending TinyDB
TinyDB Status and Roadmap
33
TinyDB Revisited
• High level abstraction:
– Data centric programming
– Interact with sensor
network as a whole
– Extensible framework
• Under the hood:
– Intelligent query processing:
query optimization, power
efficient execution
– Fault Mitigation:
automatically introduce
redundancy, avoid problem
areas
SELECT MAX(mag)
FROM sensors
WHERE mag > thresh
SAMPLE PERIOD 64ms
App
Query,
Trigger
Data
TinyDB
Sensor Network
34
Feature Overview
•
•
•
•
•
•
Declarative SQL-like query interface
Metadata catalog management
Multiple concurrent queries
Network monitoring (via queries)
In-network, distributed query processing
Extensible framework for attributes,
commands and aggregates
• In-network, persistent storage
35
Architecture
TinyDB GUI
TinyDB Client API
JDBC
PC side
Mote side
0
0
4
TinyDB query
processor
2
1
5
Sensor network
DBMS
83
6
7
36
Data Model
• Entire sensor network as one single, infinitely-long logical
table: sensors
• Columns consist of all the attributes defined in the network
• Typical attributes:
– Sensor readings
– Meta-data: node id, location, etc.
– Internal states: routing tree parent, timestamp, queue length,
etc.
• Nodes return NULL for unknown attributes
• On server, all attributes are defined in catalog.xml
• Discussion: other alternative data models?
37
Query Language (TinySQL)
SELECT <aggregates>, <attributes>
[FROM {sensors | <buffer>}]
[WHERE <predicates>]
[GROUP BY <exprs>]
[SAMPLE PERIOD <const> | ONCE]
[INTO <buffer>]
[TRIGGER ACTION <command>]
38
Comparison with SQL
• Single table in FROM clause
• Only conjunctive comparison predicates in
WHERE and HAVING
• No subqueries
• No column alias in SELECT clause
• Arithmetic expressions limited to column
op constant
• Only fundamental difference: SAMPLE
PERIOD clause
39
TinySQL Examples
“Find the sensors in bright
nests.”
Sensors
1
SELECT nodeid, nestNo, light
FROM sensors
WHERE light > 400
EPOCH DURATION 1s
Epoch Nodeid nestNo Light
0
1
17
455
0
2
25
389
1
1
17
422
1
2
25
405
40
TinySQL Examples (cont.)
2 SELECT AVG(sound)
FROM sensors
EPOCH DURATION 10s
3 SELECT region, CNT(occupied)
AVG(sound)
FROM sensors
GROUP BY region
HAVING AVG(sound) > 200
EPOCH DURATION 10s
“Count the number occupied
nests in each loud region of
the island.”
Epoch
region
CNT(…)
AVG(…)
0
North
3
360
0
South
3
520
1
North
3
370
1
South
3
520
Regions w/ AVG(sound) > 200
41
Event-based Queries
• ON event SELECT …
• Run query only when interesting events
happens
• Event examples
– Button pushed
– Message arrival
– Bird enters nest
• Analogous to triggers but events are userdefined
42
Query over Stored Data
•
•
•
•
•
Named buffers in Flash memory
Store query results in buffers
Query over named buffers
Analogous to materialized views
Example:
– CREATE BUFFER name SIZE x (field1 type1, field2
type2, …)
– SELECT a1, a2 FROM sensors SAMPLE PERIOD d INTO
name
– SELECT field1, field2, … FROM name SAMPLE PERIOD d
43
Using the Java API
• SensorQueryer
– translateQuery() converts TinySQL string into
TinyDBQuery object
– Static query optimization
• TinyDBNetwork
– sendQuery() injects query into network
– abortQuery() stops a running query
– addResultListener() adds a ResultListener that is invoked
for every QueryResult received
– removeResultListener()
• QueryResult
– A complete result tuple, or
– A partial aggregate result, call mergeQueryResult() to
combine partial results
• Key difference from JDBC: push vs. pull
44
Writing Scripts with TinyDB
• TinyDB’s text interface
– java net.tinyos.tinydb.TinyDBMain –run
“select …”
– Query results printed out to the console
– All motes get reset each time new query
is posed
• Handy for writing scripts with shell,
perl, etc.
45
Using the GUI Tools
• Demo time
46
Inside TinyDB
SELECT
T:1, AVG: 225
AVG(temp) Queries
Results T:2, AVG: 250
WHERE
light > 400
Multihop
Network
Query Processor
Aggavg(temp)
~10,000
Lines Embedded C Code
Filter
Name: temp
light >
400
got(‘temp’)
~5,000
LinesSamples
(PC-Side)
Java Time to sample: 50 uS
get (‘temp’) Tables
Cost to sample: 90 uJ
Schema
~3200 Bytes
RAM (w/ 768 byte
heap) Table: 3
Calibration
getTempFunc(…)
Units: Deg. F
TinyOS code
~58 kB compiled
Error: ± 5 Deg F
Get f Program)
: getTempFunc()…
(3x larger than 2nd largest TinyOS
47
TinyDB
Tree-based Routing
• Tree-based routing
– Used in:
• Query delivery
• Data collection
• In-network aggregation
– Relationship to indexing?
Q:SELECT …
A
Q
R:{…}
Q
R:{…}
B
Q
R:{…}Q
Q
D
R:{…}Q
C
Q
Q
R:{…}
Q
Q
Q
F
E
Q
48
Power Management Approach
Coarse-grained app-controlled communication scheduling
Mote ID
1
… zzz …
Epoch (10s -100s of seconds)
… zzz …
2
3
4
5
time
2-4s Waking Period
49
Time Synchronization
• All messages include a 5 byte time stamp indicating system
time in ms
– Synchronize (e.g. set system time to timestamp) with
• Any message from parent
• Any new query message (even if not from parent)
– Punt on multiple queries
– Timestamps written just after preamble is xmitted
• All nodes agree that the waking period begins when (system
time % epoch dur = 0)
– And lasts for WAKING_PERIOD ms
• Adjustment of clock happens by changing duration of sleep
cycle, not wake cycle.
50
Extending TinyDB
• Why extending TinyDB?
–
–
–
–
New
New
New
New
sensors  attributes
control/actuation  commands
data processing logic  aggregates
events
• Analogous to concepts in objectrelational databases
51
Adding Attributes
• Types of attributes
– Sensor attributes: raw or cooked sensor
readings
– Introspective attributes: parent,
voltage, ram usage, etc.
– Constant attributes: constant values
that can be statically or dynamically
assigned to a mote, e.g., nodeid, location,
etc.
52
Adding Attributes (cont)
• Interfaces provided by Attr component
– StdControl: init, start, stop
– AttrRegister
•
•
•
•
command registerAttr(name, type, len)
event getAttr(name, resultBuf, errorPtr)
event setAttr(name, val)
command getAttrDone(name, resultBuf, error)
•
•
•
•
•
command startAttr(attr)
event startAttrDone(attr)
command getAttrValue(name, resultBuf, errorPtr)
event getAttrDone(name, resultBuf, error)
command setAttrValue(name, val)
– AttrUse
53
Adding Attributes (cont)
•
•
Steps to adding attributes to TinyDB
1) Create attribute nesC components
2) Wire new attribute components to
TinyDBAttr configuration
3) Reprogram TinyDB motes
4) Add new attribute entries to catalog.xml
Constant attributes can be added on the
fly through TinyDB GUI
54
Adding Aggregates
• Step 1: wire new nesC components
TinyDB Aggregation Framework
Aggregate
SumM.nc
AggregateUseM.nc
stateSize(ID, ...)
merge(ID, ...)
Operator
AggregateUse
AggOperator.nc
update(ID, ...)
Aggregate
CountM.nc
hasData(ID,...)
finalize(ID,...)
init(ID, ...)
getProperties(ID)
Aggregate
AggOperatorConf.nc
AvgM.nc
55
Adding Aggregates (cont)
• Step 2: add entry to catalog.xml
<aggregate>
<name>AVG</name>
<id>5</id>
<temporal>false</temporal>
<readerClass>net.tinyos.tinydb.AverageClass</readerClass
>
</aggregate>
• Step 3 (optional): implement reader class in Java
– a reader class interprets and finalizes aggregate state
received from the mote network, returns final result as a
string for display.
56
TinyDB Status
• Latest released with TinyOS 1.1 (9/03)
– Install the task-tinydb package in TinyOS 1.1 distribution
– First release in TinyOS 1.0 (9/02)
– Widely used by research groups as well as industry pilot
projects
• Successful deployments in Intel Berkeley Lab and
redwood trees at UC Botanical Garden
– Largest deployment: ~80 weather station nodes
– Network longevity: 4-5 months
57
The Redwood Tree Deployment
• Redwood Grove in UC Botanical
Garden, Berkeley
• Collect dense sensor readings to
monitor climatic variations
across
–
–
–
–
altitudes,
angles,
time,
forest locations, etc.
• Versus sporadic monitoring
points with 30lb loggers!
• Current focus: study how dense
sensor data affect predictions
of conventional tree-growth
models
58
Data from Redwoods
Humidity vs. Time
101
104
109
110
111
36m
33m: 111
32m: 110
30m: 109,108,107
Rel Humidity (%)
95
85
75
65
55
45
35
20m: 106,105,104
Temperature vs. Time
10m: 103, 102, 101
Temperature (C)
33
28
23
18
13
8
7/7/03 7/7/03 7/7/03 7/7/03 7/7/03 7/8/03 7/8/03 7/8/03 7/8/03 7/8/03 7/8/03 7/9/03 7/9/03 7/9/03 7/9/03
9:40 13:11 16:43 20:15 23:46 3:18 6:50 10:21 13:53 17:25 20:56 0:28 4:00 7:31 11:03
Date
59
TinyDB Roadmap (near term)
• Support for high frequency sampling
– Equipment vibration monitoring, structural monitoring,
etc.
– Store and forward
– Bulk reliable data transfer
– Scheduling of communications
• Port to Intel Mote
• Deployment in Intel Fab equipment monitoring
application and the Golden Gate Bridge monitoring
application
60
For more information
• http://berkeley.intel-research.net/tinydb
or
http://triplerock.cs.bekeley.edu/tinydb
61
Part 3
Database Research Issues in Sensor
Networks
62
Sensor Network Research
• Very active research area
– Can’t summarize it all
• Focus: database-relevant research topics
– Some outside of Berkeley
– Other topics that are itching to be scratched
– But, some bias towards work that we find
compelling
63
Topics
•
•
•
•
•
•
In-network aggregation
Acquisitional Query Processing
Heterogeneity
Intermittent Connectivity
In-network Storage
Statistics-based summarization and
sampling
• In-network Joins
• Adaptivity and Sensor Networks
• Multiple Queries
64
Topics
•
•
•
•
•
•
In-network aggregation
Acquisitional Query Processing
Heterogeneity
Intermittent Connectivity
In-network Storage
Statistics-based summarization and
sampling
• In-network Joins
• Adaptivity and Sensor Networks
• Multiple Queries
65
Tiny Aggregation (TAG)
• In-network processing of aggregates
– Common data analysis operation
• Aka gather operation or reduction in || programming
– Communication reducing
• Operator dependent benefit
– Across nodes during same epoch
• Exploit query semantics to improve
efficiency!
Madden, Franklin, Hellerstein, Hong. Tiny AGgregation (TAG), OSDI 2002.
66
Basic Aggregation
• In each epoch:
– Each node samples local sensors once
– Generates partial state record (PSR)
• local readings
• readings from children
– Outputs PSR during assigned comm. interval
• At end of epoch, PSR for whole network
output at root
• New result on each successive epoch
1
2
3
4
5
• Extras:
– Predicate-based partitioning via GROUP BY
67
Illustration: Aggregation
SELECT COUNT(*)
FROM sensors
Sensor #
1
Interval #
4
2
3
Interval 4
1
4
Epoch
5
1
2
3
3
2
1
4
4
1
5
68
Illustration: Aggregation
SELECT COUNT(*)
FROM sensors
Sensor #
1
2
3
Interval #
2
1
4
4
3
Interval 3
5
1
2
3
2
2
4
1
4
5
69
Illustration: Aggregation
SELECT COUNT(*)
FROM sensors
Sensor #
1
2
3
Interval #
1
4
4
1
5
1
3
2
Interval 2
3
2
3
2
1
3
4
1
4
5
70
Illustration: Aggregation
SELECT COUNT(*)
FROM sensors
Sensor #
1
2
3
Interval #
2
3
2
2
4
5
1
3
Interval 1
1
4
4
1
5
1
3
4
5
5
71
Illustration: Aggregation
SELECT COUNT(*)
FROM sensors
Sensor #
1
2
3
Interval #
5
1
3
2
3
2
2
4
1
4
4
1
Interval 4
1
3
4
5
1
1
5
72
Aggregation Framework
• As in extensible databases, TinyDB supports any
aggregation function conforming to:
Aggn={finit, fmerge, fevaluate}
Finit {a0}
 <a0>
Partial State Record (PSR)
Fmerge {<a1>,<a2>}  <a12>
Fevaluate {<a1>}
 aggregate value
Example: Average
AVGinit
{v}
 <v,1>
AVGmerge {<S1, C1>, <S2, C2>}
 < S1 + S2 , C1 + C2>
AVGevaluate{<S, C>}
 S/C
Restriction: Merge associative, commutative
73
Taxonomy of Aggregates
• TAG insight: classify aggregates according to various
functional properties
– Yields a general set of optimizations that can automatically be
applied
Property
Partial State
Examples
MEDIAN : unbounded,
MAX : 1 record
Affects
Effectiveness of TAG
Monotonicity
COUNT : monotonic
AVG : non-monotonic
MAX : exemplary
COUNT: summary
MIN : dup. insensitive,
AVG : dup. sensitive
Hypothesis Testing, Snooping
Exemplary vs.
Summary
Duplicate
Sensitivity
Drives an API!
Applicability of Sampling,
Effect of Loss
Routing Redundancy
74
Use Multiple Parents
• Use graph structure
– Increase delivery probability with no communication overhead
• For duplicate insensitive aggregates, or
• Aggs expressible as sum of parts
– Send (part of) aggregate to all parents
SELECT COUNT(*)
• In just one message, via multicast
R
– Assuming independence, decreases variance
P(link xmit successful) = p
P(success from A->R) = p2
E(cnt) = c *
p2
Var(cnt) = c2 * p2 * (1 – p2)
V
# of parents = n
E(cnt) = n * (c/n * p2)
(c/n)2
Var(cnt) = n *
*
p2 * (1 – p2) = V/n
B
C
c
c/n
n=2
c/n
A
75
Multiple Parents Results
With Splitting
Benefit of Result Splitting
(COUNT query)
1400
1200
Avg. COUNT
No Splitting
• Better than
previous
analysis expected!
Critical
• Losses aren’t
Link!
independent!
• Insight: spreads data
over many links
1000
800
Splitting
No Splitting
600
400
200
0
(2500 nodes, lossy radio model, 6 parents per
node)
76
Acquisitional Query
Processing (ACQP)
• TinyDB acquires AND processes data
– Could generate an infinite number of samples
• An acqusitional query processor controls
– when,
– where,
– and with what frequency data is collected!
• Versus traditional systems where data is provided
a priori
Madden, Franklin, Hellerstein, and Hong. The Design of An
77
Acqusitional Query Processor. SIGMOD, 2003.
ACQP: What’s Different?
• How should the query be processed?
– Sampling as a first class operation
• How does the user control acquisition?
– Rates or lifetimes
– Event-based triggers
• Which nodes have relevant data?
– Index-like data structures
• Which samples should be transmitted?
– Prioritization, summary, and rate control
78
Operator Ordering: Interleave Sampling +
Selection
SELECT light, mag
FROM sensors
WHERE pred1(mag)
AND pred2(light)
EPOCH DURATION 1s
Traditional DBMS
(pred1)
(pred2)
At 1 sample / sec, total power savings
• could
E(sampling
mag) as
>> 3.5mW
E(sampling
be as much
 light)
1500 uJ vs.
uJ
Comparable
to 90
processor!
Correct ordering
(unless pred1 is very selective
and pred2 is not):
(pred1)
ACQP
Costly
(pred2)
Cheap
mag
light
mag
light
(pred2)
light
(pred1)
mag
79
Exemplary Aggregate
Pushdown
SELECT WINMAX(light,8s,8s)
FROM sensors
WHERE mag > x
EPOCH DURATION 1s
Traditional DBMS
WINMAX
(mag>x)
ACQP
WINMAX
(mag>x)
mag
• Novel, general
pushdown
technique
• Mag sampling is
the most
expensive
operation!
(light > MAX)
light
mag
light
80
Topics
•
•
•
•
•
•
•
•
•
In-network aggregation
Acquisitional Query Processing
Heterogeneity
Intermittent Connectivity
In-network Storage
Statistics-based summarization and sampling
In-network Joins
Adaptivity and Sensor Networks
Multiple Queries
81
Heterogeneous Sensor
Networks
• Leverage small numbers of high-end nodes
to benefit large numbers of inexpensive
nodes
• Still must be transparent and ad-hoc
• Key to scalability of sensor networks
• Interesting heterogeneities
–
–
–
–
–
Energy: battery vs. outlet power
Link bandwidth: Chipcon vs. 802.11x
Computing and storage: ATMega128 vs. Xscale
Pre-computed results
Sensing nodes vs. QP nodes
82
Computing Heterogeneity with
TinyDB
• Separate query processing from sensing
– Provide query processing on a small number of nodes
– Attract packets to query processors based on “service
value”
• Compare the total energy consumption of the
network
•
•
•
•
No aggregation
All aggregation
Opportunistic aggregation
HSN proactive
aggregation
Mark Yarvis and York Liu, Intel’s Heterogeneous Sensor
Network Project,
ftp://download.intel.com/research/people/HSN_IR_Day_Poster_03.pdf.
83
5x7 TinyDB/HSN Mica2 Testbed
84
Data Packet Saving
Data Packet Saving
0.00%
% Change in Data Packet Count
-5.00%
• How many
aggregators are
desired?
• Does placement
matter?
-10.00%
-15.00%
11% aggregators
achieve 72% of max
data reduction
-20.00%
-25.00%
-30.00%
-35.00%
-40.00%
-45.00%
-50.00%
1
2
3
4
5
6
All (35)
Number of Aggregator
Data Packet Saving - Aggregator Placement
% Change in Data Packet Counnt
0.00%
-5.00%
-10.00%
-15.00%
-20.00%
-25.00%
Optimal placement 2/3
distance from sink.
-30.00%
-35.00%
-40.00%
-45.00%
-50.00%
25
27
29
31
Aggregator Location
All (35)
85
Occasionally Connected
Sensornets
internet
TinyDB Server
GTWY
TinyDB QP
Mobile GTWY
Mobile GTWY
Mobile GTWY
TinyDB QP
GTWY
TinyDB QP
86
Occasionally Connected
Sensornets Challenges
• Networking support
– Tradeoff between reliability, power consumption
and delay
– Data custody transfer: duplicates?
– Load shedding
– Routing of mobile gateways
• Query processing
– Operation placement: in-network vs. on mobile
gateways
– Proactive pre-computation and data movement
• Tight interaction between networking and QP
Fall, Hong and Madden, Custody Transfer for Reliable Delivery in Delay Tolerant
Networks, http://www.intel-research.net/Publications/Berkeley/081220030852_157.pdf .
87
Distributed In-network Storage
• Collectively, sensornets have large amounts
of in-network storage
• Good for in-network consumption or
caching
• Challenges
– Distributed indexing for fast query
dissemination
– Resilience to node or link failures
– Graceful adaptation to data skews
– Minimizing index insertion/maintenance cost
88
Example: DIM
• Functionality
– Efficient range query for
multidimensional data.
• Approaches
– Divide sensor field into bins.
– Locality preserving mapping
from m-d space to
geographic locations.
– Use geographic routing such
as GPSR.
E2= <0.6, 0.7>
E1 = <0.7, 0.8>
• Assumptions
– Nodes know their locations
and network boundary
– No node mobility
Q1=<.5-.7, .5-1>
Xin Li, Young Jin Kim, Ramesh Govindan and Wei Hong, Distributed Index
for Multi-dimentional Data (DIM) in Sensor Networks, SenSys 2003.
89
Statistical Techniques
• Approximations, summaries, and
sampling based on statistics
• Applications:
– Limited bandwidth and large number of
nodes -> data reduction
– Lossiness -> predictive modeling
– Uncertainty -> tracking correlations and
changes over time
90
IDSQ
• Idea: task sensors in order of best
improvement to estimate of some value:
– Choose leader(s)
• Suppress subordinates
• Task subordinates, one at a time
– Until some measure of goodness (error bound) is met
» E.g. “Mahalanobis Distance” -- Accounts for
correlations in axes, tends to favor minimizing
principal axis
See “Scalable Information-Driven Sensor Querying and Routing for ad hoc Heterogeneous
Sensor Networks.” Chu, Haussecker and Zhao. Xerox TR P2001-10113. May, 2001. 91
Graphical Representation
Model location
estimate as a point
with 2-dimensional
Gaussian
uncertainty.
Residual 1
Residual 2
Area of residuals is equal
Principal Axis
S1
S2
Preferred
because it
reduces error
along principal 92
axis
In-Net Regression
• Linear regressionX :vssimple
way to predict
Y w/ Curve Fit
future12values, identify outliers
y = 0.9703x - 0.0067
10
2
• Regression can be acrossRlocal
= 0.947or remote
values, 8multiple dimensions, or with high
degree6polynomials
– E.g., node
A readings vs. node B’s
4
– Or, location
(X,Y), versus temperature
2
E.g., over many nodes
0
1
3
5
7
9
Guestrin, Thibaux, Bodik, Paskin, Madden. “Distributed Regression: an Efficient93
Framework for Modeling Sensor Network Data .” Under submission.
In-Net Regression (Continued)
• Problem: may require data from all sensors to
build model
• Solution: partition sensors into overlapping
“kernels” that influence each other
– Run regression in each kernel
• Requiring just local communication
– Blend data between kernels
– Requires some clever matrix manipulation
• End result: regressed model at every node
– Useful in failure detection, missing value estimation
94
Correlated Attributes
• Data in sensor networks is correlated; e.g.,
–
–
–
–
–
Temperature and voltage
Temperature and light
Temperature and humidity
Temperature and time of day
etc.
95
Exploiting Correlations in
Query Processing
• Simple idea:
– Given predicate P(A) over expensive attribute A
– Replace it with P’ over cheap attribute A’ such that
P’ evaluates to P
– Problem: unless A and A’ are perfectly correlated, P’
≠ P for all time
• So we could incorrectly accept or reject some readings
• Alternative: use correlations to improve
selectivity estimates in query optimization
– Construct conditional plans that vary predicate
order based on prior observations
96
Exploiting Correlations (Cont.)
•
•
•
Insight: by observing a (cheap and correlated) variable not involved
in the query, it may be possible to improve query performance
– Improves estimates of selectivities
Use conditional plans
Example
T
Light
Light>>
100
100Lux
Lux
Temp
Temp<<
20°
20°CC
Expected
ExpectedCost
Cost==150
110
Cost
Cost==100
100
Cost
Cost==100
100
Selectivity
Selectivity==.5.1 Selectivity
Selectivity==.5.9
Time in
[6pm, 6am]
F
Temp
Temp<<
20°
20°CC
Light
Light>>
100
100Lux
Lux
Cost
Cost==100
100
Cost
Cost==100
100
Selectivity
Selectivity==.5.1 Selectivity
Selectivity==.5.9
Expected
ExpectedCost
Cost==150
110
97
In-Network Join Strategies
• Types of joins:
– non-sensor -> sensor
– sensor -> sensor
• Optimization questions:
– Should the join be pushed down?
– If so, where should it be placed?
– What if a join table exceeds the memory
available on one node?
98
Choosing Where to Place
Operators
• Idea : choose a “join node” to run the operator
• Over time, explore other candidate placements
– Nodes advertise data rates to their neighbors
– Neighbors compute expected cost of running the
join based on these rates
– Neighbors advertise costs
– Current join node selects a new, lower cost node
Bonfils + Bonnet, Adaptive and Decentralized Operator
Placement for In-Network QueryProcessing IPSN 2003.
99
Topics
•
•
•
•
•
•
In-network aggregation
Acquisitional Query Processing
Heterogeneity
Intermittent Connectivity
In-network Storage
Statistics-based summarization and
sampling
• In-network Joins
• Adaptivity and Sensor Networks
• Multiple Queries
100
Adaptivity In Sensor Networks
• Queries are long running
• Selectivities change
– E.g. night vs day
• Network load and available energy vary
• All suggest that some adaptivity is needed
– Of data rates or granularity of aggregation
when optimizing for lifetimes
– Of operator orderings or placements when
selectivities change (c.f., conditional plans for
correlations)
• As far as we know, this is an open problem!
101
Multiple Queries and Work
Sharing
• As sensornets evolve, users will run many
queries simultaneously
– E.g., traffic monitoring
• Likely that queries will be similar
– But have different end points, parameters, etc
• Would like to share processing, routing as
much as possible
• But how? Again, an open problem.
102
Concluding Remarks
• Sensor networks are an exciting emerging technology,
with a wide variety of applications
• Many research challenges in all areas of computer science
– Database community included
– Some agreement that a declarative interface is right
• TinyDB and other early work are an important first step
• But there’s lots more to be done!
103