Transcript Slide 1

Data Center TCP
(DCTCP)
Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye
Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan
Modified by Feng Xie
1
Data Center Packet Transport
• Cloud computing service
provider
– Amazon,Microsoft,Google
• Transport inside the DC
– TCP rules (99.9% of traffic)
• How’s TCP doing?
2
TCP in the Data Center
• We’ll see TCP does not meet demands of apps.
– Incast
 Suffers from bursty packet drops
 Not fast enough utilize spare bandwidth
– Builds up large queues:
 Adds significant latency.
 Wastes precious buffers, esp. bad with shallow-buffered switches.
• Operators work around TCP problems.
‒ Ad-hoc, inefficient, often expensive solutions
• Our solution: Data Center TCP
3
Case Study: Microsoft Bing
• Measurements from 6000 server production cluster
• Instrumentation passively collects logs
‒ Application-level
‒ Socket-level
‒ Selected packet-level
• More than 150TB of compressed data over a month
4
Partition/Aggregate Application Structure
TLA
Picasso
Art is…
1.
Deadline
2. Art is=a250ms
lie…
…..
3.
Picasso
• Time is money
MLA ……… MLA
1.
 Strict deadlines (SLAs)
Deadline = 50ms
2.
2. The chief…
3.
…..
3.
…..
• Missed deadline
1. Art is a lie…
 Lower quality result
“It“I'd
is“Art
“Computers
“Inspiration
your
chief
like
“Bad
isto
you
awork
enemy
lie
live
artists
can
that
in
as
are
does
of
imagine
life
amakes
copy.
useless.
creativity
poor
that
exist,
man
us
is the
real.”
is
Deadline“Everything
=“The
10ms
They
but can
itultimate
with
Good
must
realize
only
good
lots
artists
find
give
seduction.“
the
of
sense.“
you
money.“
you
truth.
steal.”
working.”
answers.”
Worker Nodes
5
Workloads
• Partition/Aggregate
(Query)
• Short messages [50KB-1MB]
(Coordination, Control state)
Delay-sensitive
Delay-sensitive
• Large flows [1MB-50MB]
(Data update)
Throughput-sensitive
6
Impairments
• Incast
• Queue Buildup
• Buffer Pressure
7
Incast
Worker 1
• Synchronized mice collide.
 Caused by Partition/Aggregate.
Aggregator
Worker 2
Worker 3
RTOmin = 300 ms
Worker 4
TCP timeout
8
Queue Buildup
Sender 1
• Big flows buildup queues.
 Increased latency for short flows.
Receiver
Sender 2
• Measurements in Bing cluster
 For 90% packets: RTT < 1ms
 For 10% packets: 1ms < RTT < 15ms
9
Data Center Transport Requirements
1. High Burst Tolerance
– Incast due to Partition/Aggregate is common.
2. Low Latency
– Short flows, queries
3. High Throughput
– Large file transfers
The challenge is to achieve these three together.
10
Balance Between Requirements
High Throughput
High Burst Tolerance
Low Latency
Deep Buffers:
 Queuing Delays
Increase Latency
Shallow Buffers:
 Bad for Bursts &
Throughput
Reduced RTOmin
(SIGCOMM ‘09)
 Doesn’t Help Latency
AQM – RED:
 Avg Queue Not Fast
Enough for Incast
Objective:
Low Queue Occupancy & High
Throughput
DCTCP
11
The DCTCP Algorithm
12
Review: The TCP/ECN Control Loop
Sender 1
ECN = Explicit Congestion Notification
ECN Mark (1 bit)
Receiver
Sender 2
13
Two Key Ideas
1. React in proportion to the extent of congestion, not its presence.
 Reduces variance in sending rates, lowering queuing requirements.
ECN Marks
TCP
DCTCP
1011110111
Cut window by 50%
Cut window by 40%
0000000001
Cut window by 50%
Cut window by 5%
2. Mark based on instantaneous queue length.
 Fast feedback to better deal with bursts.
18
Data Center TCP Algorithm
B
Switch side:
– Mark packets when Queue Length > K.
Mark
K Don’t
Mark
Sender side:
– Maintain running average of fraction of packets marked (α).
In each RTT:
 Adaptive window decreases:
– Note: decrease factor between 1 and 2.
19
(Kbytes)
DCTCP in Action
Setup: Win 7, Broadcom 1Gbps Switch
Scenario: 2 long-lived flows, K = 30KB
20
Why it Works
1. High Burst Tolerance
 Large buffer headroom → bursts fit.
 Aggressive marking → sources react before packets are
dropped.
2. Low Latency
 Small buffer occupancies → low queuing delay.
3. High Throughput
 ECN averaging → smooth rate adjustments, cwind low variance.
21
Analysis
Window Size
W*+1
W*
(W*+1)(1-α/2)
Time
22
Analysis
Window Size
Packets sent in this
RTT are marked.
W*+1
W*
(W*+1)(1-α/2)
Time
22
Analysis
• How low can DCTCP maintain queues without loss of throughput?
• How do we set the DCTCP parameters?
 Need to quantify queue size oscillations (Stability).
85% Less Buffer than TCP
22
Evaluation
• Implemented in Windows stack.
• Real hardware, 1Gbps and 10Gbps experiments
–
–
–
–
90 server testbed
Broadcom Triumph
Cisco Cat4948
Broadcom Scorpion
48 1G ports – 4MB shared memory
48 1G ports – 16MB shared memory
24 10G ports – 4MB shared memory
• Numerous benchmarks
– Throughput and Queue Length
– Multi-hop
– Queue Buildup
– Buffer Pressure
– Fairness and Convergence
– Incast
– Static vs Dynamic Buffer Mgmt
23
Experiment implement
• 45 1G servers connected to a Triumph, a 10G server extern
connection
– 1Gbps links K=20
– 10Gbps link K=65
• Generate query, and background traffic
– 10 minutes, 200,000 background, 188,000 queries
• Metric:
– Flow completion time for queries and background flows.
We use RTOmin = 10ms for both TCP & DCTCP.
24
Baseline
Background Flows
Query Flows
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
 High throughput for long flows.
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
 High throughput for long flows.
 High burst tolerance for query flows.
25
Scaled Background & Query
10x Background, 10x Query
26
Conclusions
• DCTCP satisfies all our requirements for Data Center
packet transport.
 Handles bursts well
 Keeps queuing delays low
 Achieves high throughput
• Features:
 Very simple change to TCP and a single switch parameter
K.
 Based on ECN mechanisms already available in
commodity switch.
27
Congestion Control for High Bandwidth-Delay
Product Networks
D. Katabi (MIT), M. Handley (UCL), C.
Rohrs (MIT) – SIGCOMM’02
29
Basics of TCP Congestion Control
• Bandwidth-delay product
– Capacity of the “pipe” between a TCP sender and a TCP
receiver
• Congestion window (cwnd)
– Sender’s estimation of the capacity
• Additive Increase and Multiplicative Decrease (AIMD)
algorithm
– no loss:
– loss:
cwnd = cwnd + s
cwnd = cwnd – cwnd/2
Motivations
• Inadequacy of TCP, as bandwidth-delay product
increases
– Prone to instability
• regardless of AQM schemes
– Inefficient
• Fairness concern
– TCP tends to bias against long RTT flows
• Satellite links, wireless links, etc.
Design Rationale
• NOT an end-to-end approach
• Using precise congestion signaling
• Decoupling efficiency and fairness control
XCP (eXplicit Control Protocol)
• Maintains high utilization, small queues, and almost
no drops, as bandwidth/delay increases
– drop: less than one in a million packets
• Maintains good performance in dynamic
environment (with many short web-like flows)
• No bias against long RTT flows
Sender&Receiver’s Role
• Sender
– Fill the congestion header
– Update cwnd = max(cwnd + H_feedback, s)
• Receiver
– Copy H_feedback to ACK
XCP – Router’s Role
• Control Interval Estimation
– Average RTT
• Efficiency Control
– Maximize link utilization
• Fairness Control
– Achieve fairness among individual flows
Efficiency Controller (EC)
• Aggragate feedback (total H_feedback)
    d  s   Q
–
–
–
–
 , : constant value
d: control interval (average RTT)
S: spare bandwidth
Q: persistent queue size
• Stability requirement determines
 = 0.4; = 0.226
– Independent of delay, capacity and number of flows
Fairness Controller (FC)
• Achieve fairness via AIMD algorithm
–  > 0, equal throughput increment of all flows
–  < 0, throughput decrement proportional to its current
throughput
Performance Evaluation
• Simulation topology
Performance Evaluation
Bandwidth
Delay
Performance Evaluation
Mice arrival rate
Different RTT flow
The dynamics of XCP
Conclusion
• XCP provides a theoretically analysis, yet effective
approach to congestion control. It obtains excellent
performance, independent of link capacity, delay and
number of flows.
DCTCP vs. XCP
• Date center • High bandwidth-delay
• Practical
• Theoretic
43
44