Transcript Deployment of an Urban Mesh Network

Urban Mesh Networks and Protocol Behavior
under Diverse Operating Conditions
Ed Knightly
Rice University
http://www.ece.rice.edu/~knightly
Joint work with Joseph Camp, Omer
Gurewitz, Vincenzo Mancuso, Jingpu Shi
Experimental Context: Large-Scale Multi-hop Networks





Technology For All Wireless Network
Research platform: programmable and observable
Wireless ISP for region since late 2004
Over 4,000 users in 3 square kilometers
Multi-tier architecture
Ed Knightly
Access Tier

Clients (mobiles and residents) access mesh infrastructure

AP density: approximately 7 Mesh AP’s per km2
Ed Knightly
Backhaul Tier

Access nodes interconnected via backhaul tier
– Access traffic forwarded to and from gateway
– Omni directional 802.11 (b, g, or a)
Ed Knightly
Capacity Points

Gateways inject capacity into backhaul tier
– Injects capacity for sufficient Mb/sec/km2
– Continued multi-hopping would be too many users over too many hops
Ed Knightly
TFA Network Architecture and Topology
Ed Knightly

802.11b/g access and
backhaul serving 4,000
users

Point to point capacity
injection tier (802.11a and
900 MHz)

Multiple radios at gateway
nodes, single radios
elsewhere

Opportunistic and
population-driven GW
locations

99% coverage over 4 km2
when complete

Future: WARP backhaul
Challenge of Diverse Operating Conditions

Protocol behavior typically narrowly understood
– Study’s particular mathematical assumptions, ns configuration,
testbed setup
Challenges:

Design protocols that are robust to diverse:
– Channel conditions
– Asymmetries
– Topologies
– Traffic matrices

Understand protocol performance in diverse operating
conditions
Ed Knightly
Two Case Studies

Explore robustness and unexplored cross-layer
interactions driven by:
–
–
–
–

Channel conditions
Asymmetries
Topologies
Traffic matrices
Two protocols
– TCP-like congestion control over multi-hop CSMA/CA
– Control traffic
Ed Knightly
Heterogeneous Channel Conditions & Traffic Types
Heterogeneous Connectivity Set
a) High quality forwarding links (selected by routing protocol)
b) Diverse non-forwarding links (broadcast medium)
Data and Control Planes
a) Large-sized data frames
b) Small-sized control frames
1)
2)
3)
4)
Ed Knightly
Link Establishment
Routing
Congestion Control
Network Management
1 Mbps
Node Down!
5 Mbps
Homogeneous Topology
Symmetric Topology
Experimental Finding

Heterogeneous connectivity matrix +
heterogeneous traffic produces diverse
cross-layer effects

Control frames force multiplicative
degradation on data plane
control
– Overhead traffic at rate r can reduce
data throughput by up to 30 times r
– Wireless Overhead Multiplier driven
primarily by non-forwarding links
data
Ed Knightly
Diverse Overhead Effects

Experiment: measure data
throughput with and without
control overhead
Identical configuration
– TX power 200 mW, RTS
disabled, Autorate enabled


Overhead of 80 kbps (approx.
10 kbps/node)
Vastly different performance
with and without overhead
– 800 to 1800 kbps degradation
– 10-20 times injected overhead
Ed Knightly
1100 kbps 1800 kbps
800 kbps
6000
isolated
5000
Throughput (kbps)

with overhead
4000
3000
2000
1000
0
n1
n2
n3
n4
n6
TFA Backhaul Node
n7
n8
Wireless Overhead Multiplier Definition

Define WOM to quantify the effect of the bits of overhead
– O is a set of OH-injecting nodes, where o  O
– O is bits/sec of injected overhead from O
– t s→r {s,r} is saturation throughput of tx (s) and rx (r)
Ed Knightly
How to Predict WOM?
control
control
O
O
S
R
S
data
R
data

If we know link characteristics, can we predict WOM?

Classify link between emitter of control traffic and data transmitter according to
protocol-defined behavior
– Decode Transmission
– Detect Channel Activity (“carrier sense range”)
– Unable to Detect Channel Activity (hidden)
Ed Knightly
“Carrier Sense Range” Does Not Exist

In-lab measurements show
no carrier sense threshold

Set-up: 3 different cards (2
Mbps fixed modulation rate,
UDP traffic)
– Constant Noise
– External 802.11 source
heard only at transmitter
(not shown)

Throughput degradation due
to transmitter becoming deaf
to ACK
– Producing excessive
backoff
– Continues to transmit
– MAC traces taken with
Kismet
Ed Knightly
Card at TX becomes
deaf to ACK packets
WOM for Existing Link Classes

Measurements for GW’s neighbors
– Injected overhead: 10 kbps,
Autorate enabled, RTS off
Transmission Range (link o to s)
– Overhead effectively sent at base
rate (2 Mbps)
– On average, quality of TFA links
enables 11 Mbps operation

Out of Range (link o to s)
– Average WOM: 10 (high variance)
– What is causing the high variance
in WOM?
Wireless Overhead Multiplier

35
30
25
20
15
10
5
0
Transmission Range
Ed Knightly
Out of Range
TCP data traffic (1500 byte),
Autorate enabled, RTS off
Relative Link Quality of Competing Links

Same link behavior as
defined by 802.11 (unable to
carrier sense) but high
variance - why?
physical layer capture
DATA_s
DATA_s
OH
OH

Asymmetric WOM with
forwarding link differences
– 2 dB difference in link quality
significantly alters
performance
Wireless Overhead Multiplier
– Same injected overhead and
non-forwarding links
– Expect high WOM values
(low variance)
14
12
link 1 < link 2
link 1 > link 2
10
8
6
4
2
0
-5 -4 -3 -2 -1
0
1
2
3
4
5
Relative SNR (link 1 - link 2)
Ed Knightly
UDP data traffic (1500 byte),
Autorate disabled, RTS off
6
7
WOM and Diverse Link Characteristics

Control traffic can have widely assumed modest effect

Link asymmetry yields severe degradation
– Overhead rate has up to 30 times impact on data rate

Strong nodes transmitting control traffic have wide
reaching effects

In a realistic topology, a mix is inevitable
Ed Knightly
Two Case Studies

Explore robustness and unexplored cross-layer interactions
driven by:
–
–
–
–

Channel conditions
Asymmetries
Topologies
Traffic matrices
Two protocols
– TCP-like congestion control over multi-hop CSMA/CA
– Control traffic
Ed Knightly
Congestion Control


Single TCP flow over multi-hop chain and 802.11
Yields high utilization with an appropriate congestion window
– Ideal window is a function of chain length
TCP DATA
A
Ed Knightly
B
GW
TCP ACK
However…

Radically different behavior for a changed traffic matrix:
– starvation arises even with fixed sliding-window flow control
coupled with CSMA (including 1!)
– individual or aggregate one-hop flows can starve multi-hop
flows

This is the basic scenario for a mesh network
– multi-hop and multi-flow
TCP DATA
A
Ed Knightly
B
GW
TCP ACK
Severe Throughput Imbalance
TCP DATA
Experiment of potential for starvation in
operational mesh networks
• inject traffic from A and B to GW
• saturation conditions
5000
A
B
TCP ACK
RTS/CTS Disabled
5000
RTS/CTS Enabled
Throughput
TCP Throughput
(Contention)
3000
2000
1000
0
4000
Throughput
TCP Throughput
(Contention)
3000
2000
1000
0
B->GW
Ed Knightly
Achievable TCP
Throughput (kbps)
Throughput (kbps)
Achievable TCP
4000
GW
Flow
A->GW
B->GW
Flow
The two-hop node “starves” when
contending with the one-hop node
A->GW
Origins of Starvation
Compounding effect of three factors:
(i)
Collision avoidance in the medium access protocol induces
bi-stability in which pairs of nodes symmetrically alternate in
capturing system resources
(ii)
Congestion control in the transport protocol induces
asymmetry in the time spent in each state and favors the
one-hop flow
(iii) High penalty due to cross-layer effects in terms of loss,
delay, and consequently, throughput, in order to re-capture
system resources
Ed Knightly
Origins (I): Medium Access and Bi-Stability
A
B
DATA
CW=2CWmin
Aggregate ACK
GW
CW=22CWmin
CW=2CWmin
CW=22CWmin
CW=CWmin
CW=2kCWmin
CW=CWmin
CW=CWmin
CW=CWmin
CW=CWmin
Due to lack of coordination:



Bi-stable state: either A transmits and GW is in high backoff, or GW
transmits and A is in high backoff
Success state and fail state alternate
Symmetric behavior
Ed Knightly
Middle Node Shares with Winner
A
B
Multiple packet burst (GW,B)
GW
Multiple packet burst (A,B)
GW traffic
A traffic
B traffic

B is in range of both A and GW (complete channel state)

B's packets interleave with A's and GW's packets
Ed Knightly
Origins (II): Asymmetry Induced by TCP
• Two nested transport loops and sliding windows
DATA
DATA
A
B
Outer loop
Inner loop
GW
ACK
ACK
• Asymmetric impact of multipacket capture: transport
loops change the duration of states
• (A, B) burst:
the burst size is limited by:
• TCP window size
• (GW, B) burst:
self-sustaining loop:
• TCP ACK are generated
Ed Knightly
DATA
DATA
A
B
GW
B
GW
DATA
A
ACK
Origins (III): Severe State Transition Penalties
• Asymmetric impact of multipacket capture
• Node GW incurs small penalty:
short duration of fail state but long packet bursts
• Node A incurs high penalty:
long duration of fail state and low offered load
high backoff & multiple TCP timeouts
TCP ACK received
TCP sequence number [kB]
335
(Cumulated ACK)
330
TCP Congestion
Window
325
MAC Packet drop
(Max Retry Limit reached)
320
315
310
TCP Timeouts
305
Node A
300
A
B
First time segment is transmitted
TCP retransmissions
295
80
Ed Knightly
85
90
Time [sec]
95
100
105
GW
Analytical Model

Objectives
– Isolate and capture the root cause of starvation
– Model one aspect of congestion control (sliding
window), queues, and CSMA/CA

Technique
– Embedded Markov chain model
– Queue state, congestion window, contention
window, carrier sense
Ed Knightly
Evaluation: Model, Simulations, and TFA

Model
– static sliding window
congestion control
3
Simulation
– fixed TCP congestion
window + timeouts,
cumulative ACKS, …
– legacy TCP New Reno
(dynamic congestion
window)

Measurements at TFA
NS2-Fixed TCP win.
NS2
Throughput (Mbps)

Model
TFA
2
1
– TCP New Reno+802.11
0
A->GW
B->GW
Model predicts starvation: experimental factors exacerbate
Ed Knightly
What Next?

Starvation exists

Understand origins through analysis and modeling
of protocols
– CSMA + sliding window are sufficient

Can we fix it?
Ed Knightly
Solution: Redistributing Queues via MAC

Recall origins
– MAC bi-stability
– Flow-control induced asymmetry
– Severe state transition penalty

A
B
Mitigate MAC bi-stability
– Re-distribute the “distributed queue”
– Decrease the steady state probability of system states where
QA > 0 and QGW > 0
QA

A
B
GW
QGW
Model-driven topology-based solution
– Middle node should access medium less aggressively to shift
queues to itself
– Increase CWmin for node B
Ed Knightly
GW
Conclusions

Urban-scale experiments preclude a narrow set of
assumptions

Reality is diversity in channels, topologies, traffic
matrices, … yielding vastly different protocol
behaviors

Challenges
– understanding protocols in diverse operating conditions
– robust protocol design
http://networks.rice.edu
Ed Knightly