Transcript Distributed Virtual-Time Scheduling in Rings (DVSR)

Distributed Virtual-Time
Scheduling in Rings (DVSR)
Chun-Hung Chen
2004.04.30
National Taipei University of
Technology
Outlines
RPR Recall
Problems in RPR
Ring Ingress Aggregated with Spatial
Reuse Fairness (RIAS)
Distributed Virtual-Time Scheduling in
Rings (DVSR)
Simulation Results
Conclusions
RPR Recall
RPR stands for Resilient Packet Ring, which is
in IEEE 802.17 Draft State
Dual rings structure with Destination strip
mechanism
Traffic is classified in three classes:

Class A (A0 or A1), Class B (CIR or EIR),
Class C
When congested, the station will compute
its approximation fair rate by


Dividing the available bandwidth between all
upstream stations that are currently sending
frames through this station
Using its own current add rate
Two operation mode

Conservative Mode
Congested station will wait a FRTT to send a new
fair rate if it is still in congestion

Aggressive Mode
Congested station sends new fair rate in every
100μs if it is still in congestion
Problems in RPR
Single Rate Controller

Per-destination rate controller is optional
Permanent Oscillation With Unbalanced Constant-Rate
Traffic Inputs


Unbalanced traffic will trigger severe and permanent oscillations
Computed add_rate or Capacity/Active_Stations do not reflect
the true situation
Throughput Loss

Utilization degrades due to oscillation
AM & CM
Convergence

Slow convergence time
Transit traffic has priority over ingress “station” traffic
Each node measures my_rate of ingress traffic
If a node is congested:


…
send my_rate upstream
upstream nodes throttle to my_rate
allow_rate
…
my_rate
congested
Throttle
traffic
Throttle traffic
After
approximately
500 iterations,
and propagate
converges to the rate
fairupstream
rates in this case
The Problem with Darwin
my_rate is NOT the ring-wide fair rate
Example of permanent oscillation and
throughput degradation in Darwin:
flow (1,3)
Input Rate = 622 Mbps
flow (2,3)
Input Rate = 1 Mbps
1
2
3
6
5
4
Modeling RPR Oscillations
(Analytical and Simulation Results)
Model accurately matches simulation
flow (1,3)
Input rate = 622 Mbps
flow (1,3)
Input rate = 1 Mbps
flow (2,3)
Input rate = 1 Mbps
...
1
2
Aggressive Mode
flow (2,3)
Input rate = 622 Mbps
3
...
...
1
2
3
Conservative Mode
...
RIAS
Ring Ingress Aggregated with Spatial Reuse
Fairness


Define the level of traffic granularity for fairness
determination at a link as an ingress-aggregated (IA)
flow
Ensure maximal spatial reuse subject to the first
constraint
Steps of RIAS



Allocate bandwidth on each link locally fair according
to an ingress aggregated granularity (IA traffic)
Refine bandwidth allocation for each IA flow
according to its egress point and bottlenecks
Reclaim unused bandwidth fairly by iterating
Highly Similar to Max-Min Flow Control
Comparison
Proportional Fair Allocation


Penalizes flows farther away from the
destination
Important for TCP in the Internet (rate
decrease with RTT)
Fairness with Ingress-Egress flow
granularity

Incorrectly rewards nodes for spreading out
traffic to many destination versus all to hub
node
Illustration of RIAS Fair (1/3)
1/4
1/4
1/4
1/4
Parking Lot

4 flows each receive rate ¼
Illustration of RIAS Fair (2/3)
3/4
1/4
1/4
1/4
1/4
Parallel Parking Lot
Each flow receives rate ¼ on downstream link
 Left 1-hop flow fully reclaims excess
bandwidth (RIAS)

Illustration of RIAS Fair (3/3)
1/2
1/4
1/4
1/4
1/4
1/4
3/4 1/2
Upstream Parallel Parking Lot

Key points:
Flow granularity for fairness
Spatial reuse
Proportional Fair
“Proportional fairness”
Penalizes flows farther away from the hub
 Important for TCP in the Internet (rate
decreases with RTT)
 TCP/GigE approximates this in the parking lot

.12
.16
.24
.48
•
Variants of all of these have been discussed and proposed in
the RPR standard meetings
Ingress-Egress Flow Granularity
Fairness with Ingress-Egress flow granularity



Incorrectly rewards nodes for spreading out traffic to
many destinations vs. all to hub node
Wrong flow granularity counts 6 flows and gives rate 1/6
(RIAS-fair: all green flows together get ¼ vs ½)
DVSR
Nodes construct a proxy of virtual time at
the ingress-aggregated flow granularity

Using per-ingress byte counts
The proxy is a lower bound on virtual time
temporally aggregated over time and
spatially aggregated over traffic flows
sharing the same ingress point (IA flows)
Distributed Fair Bandwidth
Allocation
Remote Fair Queuing


Control of upstream rate controllers via use of
ingress-aggregated virtual time as a congestion
message received from downstream nodes
Conceptually an ideal GPS processor
Delayed and Temporally Aggregated Control
Information

Proxy of Virtual Time
Multinode RIAS Fairness

Three Steps to approximate RIAS
Remote Fair Queuing:
Single Resource Illustration
Rate Controller
...
...
...
...
MUX
GPS
v(t)
Feedback
Delay 
(a) GPS Server
•
(b) Approximation
Control of upstream rate controllers via
downstream virtual time progression
• True fair queueing replaced with rate controllers
+ multiplexer
• Note: no packets queued in mux when  = 0
Example
Link capacity = 1
pkt/sec
T = 10 pkt
transmission times
b = 0.8 (fraction of time
busy)
Limiter value = 0.8
Packet Size
Packet Size
1
1
0
1
2
3
4
5
t
0
(a) Traffic Arrival for Flow 1
1
2
3
t
4
(b) Traffic Arrival for Flow 2
v(t)
6
5
4
3
2
>0

Controller set at t for rates in
[t-T- , t- ]
1
0
1
2
3
4
5
6
7
(c) Virtual Time
8
9
10
t
Step I: Local Fairness
Label nodes 1, …, N and links 1, …, N-1
rij is the traffic demand between nodes i and j at a
particular time instant
rin is the Ingress Aggregated traffic from ingress
node i at link n
rin = ∑j>nrij
The locally fair allocation on link n is
Rin = max_mini(C,r1n,r2n,…,rin,…, rnn)
Footnote on max_min
What is max_mini( )?
The “textbook” definition of (locally) fair
 Would be achieved by fair queueing if fair
queueing was performed on ingress
aggregates
 Can write down the exact computation
[BerGal92,p527]
 Maximizing the network use allocated to the
sessions with the minimum allocation

Step II: Ingress Fairly Sub-allocates
Per-link Bandwidths
•
Rijn = max_minj(Rin,ri,n+1,ri,n+2,…,ri,j,…,ri,N)
•
Ingress has bandwidth Rin on link n and divides it
fairly among flows traversing n
•
End-to-End rate is the bottleneck rate
ri,j = minnRijn, n=i, i+1,…,j-1
Step III: Iterate
•
There may be further bandwidth available for
spatial reuse
– Due to multiple congestion points
•
Iterate process such that all excess capacity is
fairly reclaimed
•
Set new capacity to all unallocated capacity
Cn=Cn-∑ijRijn
Go to Step I
•
DVSR Protocol
Scheduling of Station versus Transit Packets


FIFO queue
Class A is not taken in consideration
Feedback Signal Computation
Feedback Signal Transmission


Control message is N bytes while there exist N
stations
Each station i writes its value at i bytes
Rate Limit Computation

Suballocate its per-link fair rates to the flows with
different egress nodes
DVSR Protocol
Scheduling

FIFO (or SP)
Computation of
feedback signal


Byte count for each
ingress node - lower
bound of virtual time
Order such that
l1 ≤ l2 ≤ … ≤ lk
Analysis of DVSR
Fairness Bound

Lemma 1
A node-backlogged flow in DVSR can be under-throttled by
at most (1-(1/N))CT

Lemma 2
A node-backlogged flow in DVSR can be over-throttled by at
most (1-(1/N))CT

Lemma 3
The service difference during any interval for two flows i and j
with infinite demand is bounded by 2(C-(1/N)C)T under
DVSR
Simulations Results
Fairness and Spatial Reuse




Fairness in the Parking Lot
Performance Isolation for TCP Traffic
RIAS versus Proportional Fairness for TCP
Traffic
Spatial Reuse in the Parallel Parking Lot
Convergence Time Comparison
Fairness in the Parking Lot
flow (1,5)
DVSR provides RIAS fair
shares
GigE does not
flow (3,5)
flow (4,5)
1
2
3
4
5
10
9
8
7
6
0.6
Normalized Throughput
Four constant-rate UDP
flows sending at 622
Mbps
flow (2,5)
DVSR
0.5
311
GigE
0.4
0.3
155.5
155.5
155.5 155
155.5
flow (3,5)
flow (4,5)
0.2
78
78
0.1
0
flow (1,5)
flow (2,5)
Flow
Spatial Reuse in the Parallel
Parking Lot
flow (1,5)
flow (3,5)
flow (4,5)
flow (1,2)
1
10
link 1
2
3
link 2
9
DVSR is within 1% of RIAS
fair rates
GigE favors downstream
flows & cannot achieve
spatial reuse
Darwin achieves only if
using “multi-choke” option
4
link 3
8
5
link 4
7
6
0.8
464.5
0.7
Normalized Throughput
CBR UDP flows
sending at the
link capacity
flow (2,5)
DVSR
0.6
GigE
310.5
0.5
310.5
0.4
0.3
157.5
154
0.2
0.1
155 155.5
155.5
104
52
0
flow (1,5) flow (2,5) flow (3,5) flow (4,5) flow (1,2)
Flow
Upstream Parallel Parking Lot
(Results in Unbalanced Traffic Even with Balanced Inputs)
flow (2,6)
flow (1,3)
...
1
2
flow (3,6)
3
flow (4,6)
4
flow (5,6)
5
Darwin oscillation range is
0.25 to 0.75 and throughput
loss is 14%
Many other scenarios can
result in traffic imbalances
and throughput losses
DVSR within 0.1% of RIAS
Darwin Behavior
6
...
RIAS vs. Proportional Fairness
for TCP Traffic
flow (1,5)
Each flow =1 TCP
micro flow (ftp/TCP
Reno)
flow (2,5)
flow (3,5)
flow (4,5)
1
2
3
4
5
10
9
8
7
6
0.35
201.5
Rate within 1% of
RIAS fair rates for 1
TCP micro-flow
GigE tends to provide
“proportional fair” rates
Normalized Throughput
DVSR
GigE
0.3
0.25
156
152.5
150.5
161.5
163
138.5
0.2
120.5
0.15
0.1
0.05
0
flow (1,5)
flow (2,5)
Flow
flow (3,5)
flow (4,5)
Convergence Time in the Parking
Lot
DVSR




Gandalf
CBR UDP flows with rate 0.4 (248.8Mbps)
Flow(1,5), (2,5), (3,5), (4,5) begin transmission at times 0.0, 0.1,
0.2, and 0.3 seconds respectively
Convergence time 0.2 msec for DVSR, 50 msec for Darwin
Richer feedback signal allows faster convergence
Inter-Node Performance
Isolation of TCP/UDP Traffic
Flow (1,5) TCP micro-flows
flow (1,5)
Others are CBR UDP flows
with rate 0.3
More TCP micro-flows –
DVSR able to achieve RIAS
fairness
Darwin performance
unknown (MAC sim
incompatible with TCP)
flow (2,5)
flow (3,5)
flow (4,5)
1
2
3
4
5
10
9
8
7
6
Conclusions
Link capacity does not be considered in
RPR
Do my_rate and forward_rate in RPR fit
the bandwidth allocation?
DVSR approximate RIAS quicker than
RPR
RPR may have better performance if
feedback mechanism is modified
Reference
V. Gambiroza, P. Yuan, B. Balzano, Y. Liu,
S.Sheafor, “Design, Analysis, and
Implementation of DVSR: A Fair HighPerformance Protocol for Packet Rings”,
IEEE/ACM Transactions on Networking, Feb.
2004
F. Davik, M.Yilmaz, S. Gjessing, N. Uzun, “IEEE
802.17 Resilient Packet Ring Tutorial”, IEEE
Communicaion Magazine, Mar. 2004
http://www.ece.rice.edu/networks/RPR/