Transcript Data Center TCP (DCTCP) Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan Microsoft Research Stanford University Presented.

Data Center TCP
(DCTCP)
Mohammad Alizadeh, Albert Greenberg, David A. Maltz, Jitendra Padhye
Parveen Patel, Balaji Prabhakar, Sudipta Sengupta, Murari Sridharan
Microsoft Research
Stanford University
Presented by Shaddi Hasan
1
Case Study: Microsoft Bing
• Measurements from 6000 server production
cluster
• Instrumentation passively collects logs
‒ Application-level
‒ Socket-level
‒ Selected packet-level
• More than 150TB of compressed data over a
month
2
Partition/Aggregate Application Structure
TLA
Picasso
Art is…
1.
Deadline
2. Art is=a250ms
lie…
…..
3.
Picasso
• Time is money
MLA ……… MLA
1.
 Strict deadlines (SLAs)
Deadline = 50ms
2.
2. The chief…
3.
…..
3.
…..
• Missed deadline
1. Art is a lie…
 Lower quality result
“It“I'd
is“Art
“Computers
“Inspiration
your
chief
like
“Bad
isto
you
awork
enemy
lie
live
artists
can
that
in
as
are
does
of
imagine
life
amakes
copy.
useless.
creativity
poor
that
exist,
man
us
is the
real.”
is
Deadline“Everything
=“The
10ms
They
but can
itultimate
with
Good
must
realize
only
good
lots
artists
find
give
seduction.“
the
of
sense.“
you
money.“
you
truth.
steal.”
working.”
answers.”
Worker Nodes
3
Workloads
• Partition/Aggregate
(Query)
• Short messages [50KB-1MB]
(Coordination, Control state)
• Large flows [1MB-50MB]
(Data update)
Delay-sensitive
Delay-sensitive
Throughput-sensitive
4
Impairments
• Incast
• Queue Buildup
• Buffer Pressure
5
Incast
Worker 1
• Synchronized mice collide.
 Caused by Partition/Aggregate.
Aggregator
Worker 2
Worker 3
RTOmin = 300 ms
Worker 4
TCP timeout
6
MLA Query Completion Time (ms)
Incast Really Happens
• Requests areth jittered over 10ms window.
99.9 off
percentile
is beinghigh
tracked.
Jittering trades
median against
percentiles.
• Jittering switched off around 8:30 am.
7
Queue Buildup
Sender 1
• Big flows buildup queues.
 Increased latency for short flows.
Receiver
Sender 2
• Measurements in Bing cluster
 For 90% packets: RTT < 1ms
 For 10% packets: 1ms < RTT < 15ms
8
Data Center Transport Requirements
1. High Burst Tolerance
– Incast due to Partition/Aggregate is common.
2. Low Latency
– Short flows, queries
3. High Throughput
– Continuous data updates, large file transfers
The challenge is to achieve these three together.
9
Tension Between Requirements
High Throughput
High Burst Tolerance
Low Latency
Deep Buffers:
 Queuing Delays
Increase Latency
Shallow Buffers:
 Bad for Bursts &
Throughput
Reduced RTOmin
(SIGCOMM ‘09)
 Doesn’t Help Latency
AQM – RED:
 Avg Queue Not Fast
Enough for Incast
Objective:
Low Queue Occupancy & High
Throughput
DCTCP
10
The DCTCP Algorithm
11
Nugget time
1. TCP causes problems because it tries to fill buffers
– Congestion is signaled by loss  buffer overflow.
2. Use ECN to signal congestion, not loss
– Keeps buffers short, reduces retransmissions
12
Review: The TCP/ECN Control Loop
Sender 1
ECN = Explicit Congestion Notification
ECN Mark (1 bit)
Receiver
Sender 2
13
Small Queues & TCP Throughput:
The Buffer Sizing Story
• Bandwidth-delay product rule of thumb:
– A single flow needs
buffers for 100% Throughput.
Cwnd
Buffer Size
B
Throughput
100%
17
Small Queues & TCP Throughput:
The Buffer Sizing Story
• Bandwidth-delay product rule of thumb:
– A single flow needs
buffers for 100% Throughput.
• Appenzeller rule of thumb (SIGCOMM ‘04):
– Large # of flows:
is enough.
Cwnd
Buffer Size
B
Throughput
100%
17
Small Queues & TCP Throughput:
The Buffer Sizing Story
• Bandwidth-delay product rule of thumb:
– A single flow needs
buffers for 100% Throughput.
• Appenzeller rule of thumb (SIGCOMM ‘04):
– Large # of flows:
is enough.
• Can’t rely on stat-mux benefit in the DC.
– Measurements show typically 1-2 big flows at each server, at most 4.
17
Small Queues & TCP Throughput:
The Buffer Sizing Story
• Bandwidth-delay product rule of thumb:
– A single flow needs
buffers for 100% Throughput.
• Appenzeller rule of thumb (SIGCOMM ‘04):
– Large # of flows:
is enough.
• Can’t rely on stat-mux benefit in the DC.
– Measurements show typically 1-2 big flows at each server, at most 4.
Real Rule of Thumb:
Low Variance in Sending Rate → Small Buffers Suffice
B
17
Two Key Ideas
1. React in proportion to the extent of congestion, not its presence.
 Reduces variance in sending rates, lowering queuing requirements.
ECN Marks
TCP
DCTCP
1011110111
Cut window by 50%
Cut window by 40%
0000000001
Cut window by 50%
Cut window by 5%
2. Mark based on instantaneous queue length.
 Fast feedback to better deal with bursts.
18
Data Center TCP Algorithm
B
Switch side:
– Mark packets when Queue Length >
K.
Sender side:
Mark
K Don’t
Mark
– Maintain running average of fraction of packets marked (α).
In each RTT:
 Adaptive window decreases:
– Note: decrease factor between 1 and 2.
19
(Kbytes)
DCTCP in Action
Setup: Win 7, Broadcom 1Gbps Switch
Scenario: 2 long-lived flows, K = 30KB
20
Why it Works
1.High Burst Tolerance
 Large buffer headroom → bursts fit.
 Aggressive marking → sources react before packets are
dropped.
2. Low Latency
 Small buffer occupancies → low queuing delay.
3. High Throughput
 ECN averaging → smooth rate adjustments, low variance
21
Baseline
Background Flows
Query Flows
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
 High throughput for long flows.
25
Baseline
Background Flows
Query Flows
 Low latency for short flows.
 High throughput for long flows.
 High burst tolerance for query flows.
25
Conclusions
• DCTCP satisfies all our requirements for Data
Center packet transport.
 Handles bursts well
 Keeps queuing delays low
 Achieves high throughput
• Features:
 Very simple change to TCP and a single switch
parameter.
 Based on mechanisms already available in Silicon.
27
Thoughts
• Convergence times?
• Actually requires persistent connections
– Section 4.2.2: “All communication is over longlived connections, so there is no three-way
handshake for each request.”
• TCP will blow DCTCP away if sharing a buffer
27
Thoughts
• This is basically the right answer for handling
incast and meeting deadlines
– Use ECN to keep queues short, rather than filling up your buffers and
waiting for loss.
– D3 incorporates explicit knowledge of deadlines, but this complicates
API and necessitates more changes.
– Complementary to MPTCP (or any other TCP for that matter)
28
Scaled Background & Query
10x Background, 10x Query
26
Thoughts
• “The key contribution here is […] the act of
deriving multi-bit feedback from the
information present in the single-bit sequence
of marks.”
– Calculating a moving average is not a contribution.
– Using ECN + clever scheme for adjusting CWIN is!
31
Analysis
• How low can DCTCP maintain queues without loss of throughput?
• How do we set the DCTCP parameters?
 Need to quantify queue size oscillations (Stability).
85% Less Buffer than TCP
22