Transcript Tight SLA Tutorial

Deploying Tight-SLA
services on an IP
Backbone
Clarence Filsfils – [email protected]
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
1
Objective
• To present design & deployment good
practices to enable tight SLAs to be
offered
– when to use what and how
– validation results
– operational guidelines
– deployment experience
• Focus on the backbone design
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
2
An overview of the Analysis
LLJ:Loss/Latency/Jitter
DiffServ
TE
Convergence
ISIS Sub-Second
FRR Sub-100ms
DSTE
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
3
Further information
• “Engineering a Multiservice IP backbone to
support tight SLAs”, Computer Networks
Special Edition on the New Internet
Architecture
• Full-Day Tutorial
–RIPE41, APRICOT 2002:
www.ibb.net/~filsfils
• Low-Level Design Guides, Validation Results
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
4
Agenda
• Introduction and SLA
• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
5
Typical Core Per Class SLA
Characteristics
Class
Through- Avail- Loss
Delay Jitter
put
ability rate



VoIP
Bus
BE






?

Typically more Classes at the Edge
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
6
One-Way Jitter
• Delay variation generally computed as the
variation of the delay for two consecutive
packets
• Due to variation of
– Propagation delay
– Switching / processing delay
– Queuing / scheduling delay
• Jitters buffers remove variation but
contribute to delay
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
9
Backbone VoIP Jitter Budget
• Typical jitter budget:
– Mouth to ear budget
100ms
– Backbone propagation
– 30ms
– Codec delay
– ~35ms
– Jitter Budget
= 35ms
> 30ms for the access
> 5ms for the core
> 10 hops => 500 µs/hop
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
10
Per flow sequence preservation
• Best-practise IP Design: per-flow loadbalacing!
• Re-ordering Impact on Service Perception
– Long-Lived TCP: degraded goodput
– Real-time video: loss rate += OOS_rate
– VoIP: jitter
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
15
Re-ordering Impact on Service
Server to Multiple Clients
Pewrcentage of applicatiosn
throughput
100
80
Linux 15ms
Unix 15 ms
Linux 35ms
Unix 35ms
60
40
20
0
0.01%
0.10%
1.00%
10.00
%
100.00
%
Rate of packets reordered
•
[LAOR01]: “Results show that packet reordering, by at least three packet locations,
of only a small percentage of packets in the backbone link can cause a significant
degradation of applications throughput. Long flows are affected the most. Due to the
potential effect, minimizing packet reordering, as well as mitigating its effect
algorithmically, should be considered”.
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
16
Loss of Connectivity / Convergence
• Incentive to reduce the loss of
connectivity (LoC)
• Availability
– 99.999% per day  0.9sec of downtime
• VoIP
– 40msec LoC: glitch
– 1, 2 sec LoC: call drop
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
17
How to specify the target for the
metric
• SLA statistical definitions do matter
– min/avg/max versus percentile
– Measured time interval…
• SLAs definitions today tend to be loose
– averaged over a month
– averaged over many POP-to-POP pairs
(temptation to add short pairs to reduce
average…)
• IP Performance Metrics IETF WG
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
18
Optimizing the IP Infrastructure
• Loss, Latency, Jitter: iif Demand < Offer
– OverProvisioned Backbone
– Differentiated Services
– Capacity Planning
– TE and DS-TE
• Loss of connectivity due to link/node
failure
– IGP Convergence
– MPLS FRR Protection
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
19
Agenda
• Introduction and SLA
• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
20
Loss of Connectivity
• IGP Backbone Convergence:
– the time it takes for connectivity to be
restored upon link/node failure/addition for
an IP flow starting on an edge access
router and ending on another edge access
router, excluding any variation of BGP
routes.
• For this session, IGP = ISIS
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
21
Historical ISIS Convergence
• 10 to 30 seconds
• Not excellent
• In the past, focus has been more on
stability than on fast convergence
– typical trade-off
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
22
What this presentation will explain
ms
IGP Backbone Convergence
9000
8000
7000
6000
5000
4000
3000
2000
1000
0
default
fast isis
• ISIS Convergence in 1 or 2 second is
conservative
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
23
Link-State protocol
overview
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
24
An example network
H
3
G
5
5
3
F
12
12
4
2
C
D
3
E
3
8
7
S2
4
B
3
S3
S1
A
3
S0
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
25
The Final SPT rooted at A
G: oif so & s3, Cost 13
5
F: oif so & s3, Cost 8
2
C: oif so & s3, Cost 6
E: oif so, Cost 11
D: oif s3, Cost 3
3
3
3
8
S3
B: oif so, Cost 3
A: oif null, Cost 0
3
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
S0
26
G: oif so & s3, Cost
13
5
F: oif so & s3, Cost 8
2
C: oif so & s3, Cost 6
E: oif so, Cost 11
G
5
3
8
5
3
S3
3
2
C
3
7
8
4
3
S
1
S0
D
3
E
A: oif null, Cost 0
12
12
B
3
S
2
G: oif s3, Cost 13
5
S
3
A
F: oif s3, Cost 8
S
0
2
C: oif s3, Cost 6
E: oif s1 &
s3, Cost 12
S3
B: oif s1, Cost 4
4
© 2002, Cisco Systems, Inc. All rights reserved.
3
D: oif s3,
Cost 3
3
8
Clarence Filsfils – Nanog 25
3
B: oif so, Cost 3
F
4
D: oif s3,
Cost 3
A: oif null, Cost 0
S1
27
The RIB construction
Lo0: 1.1.1.1/32, C=0
Pos1: 2.0.0.1/30, C=2
C: oif so & s3, Cost 6
D: oif s3, Cost 3
3
3
3
S3
B: oif so, Cost 3
A: oif null, Cost 0
3
S0
• ISIS adds the following paths to the RIB:
– 1.1.1.1/32: OIF = S0 or S3 with Metric 6 (6+0)
– 2.0.0.1/30: OIF = S0 or S3 with Metric 8 (6+2)
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
28
LSDB, RIB and FIB
sh isis data
ISIS LSDB
Static
Routes
BGP table
Best
RIB
sh ip route
FIB & dFIB
sh ip cef
Control
Data Plane
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
29
SPF optimisations
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
30
SPF Optimizations
• Most Basic Implementation
– Any change (link, node, leave)
 recompute the whole SPT and the whole RIB
• Optimization 1: decouple SPT and RIB
Called “SPF”
– If any topology change (node, link)
 recompute SPT and the RIB
– If only a leave change (IP prefix)
Called “PRC”
 keep the SPT, just update the RIB for the
nodes whose leaves have changed
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
31
PRC
G
Int lo 0: 65.1.1.1/32
F
Cost: 8, NH: D, B
C
E
Cost: 13, NH: D
Cost: 6, NH: D, B
D
Cost: 3, NH: D
Cost: 11, NH: B
S2
B
Cost: 3, NH: B
S3
S1
S0
A
Cost: 0, NH: --
• PRC here consists in just adding 65.1.1.1/32
in the RIB. The SPT is not affected.
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
32
Incremental-SPF
• Optimization 2
• When the topology has changed, instead
of building the whole SPT from scratch
just fix the part of the SPT that is affected
• Only the leaves of the nodes re-analyzed
during that process are updated in the RIB
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
33
Incremental-SPF
G
F
Cost: 8, NH: D, B
C
E
Cost: 13, NH: D
Cost: 6, NH: D, B
D
Cost: 3, NH: D
Cost: 11, NH: B
S2
B
Cost: 3, NH: B
S0
Clarence Filsfils – Nanog 25
S3
S1
© 2002, Cisco Systems, Inc. All rights reserved.
A
Cost: 0, NH: --
C-G link is down.
C-G link was not used in SPT
anyway, therefore there is no
need to run SPF.
34
Incremental-SPF
G
H
F
Cost: 8, NH: D, B
C
E
F reports a new neighbor.
The SPT need only to be
extended behind F. There is
no need for router A to
recompute the whole SPT
Router A will compute SPF
from node F
Clarence Filsfils – Nanog 25
Cost: 13, NH: D
Cost: 6, NH: D, B
D
Cost: 3, NH: D
Cost: 11, NH: B
S2
B
Cost: 3, NH: B
© 2002, Cisco Systems, Inc. All rights reserved.
S3
S1
S0
A
Cost: 0, NH: --
35
Incremental-SPF
• More information is kept in the SPT
–Parents list
–Neighbors list
• Based on the changed information, the
SPT is “modified” in order to reflect the
changes
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
36
Incremental-SPF
• The further away from the root the change,
the higher the gain
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
37
SPF, PRC, I-SPF: summary
• Only a leaf change
– PRC
• Graph impacted
– normal-SPF: recompute the full SPT and
hence reinserts all the ISIS routes in the
RIB
– I-SPF: only recomputes the part of the
SPT that is affected. Only the leaves from
that part are affected.
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
38
Topology and Leaf
Optimizations
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
39
Parallel point-to-point adjacencies
C
D
3
E
3
8
3
S2
4
B
LSP B
IS: 3 A
IS: 4 A
IS: 3 C
IS: 8 E
7
S3
S1
A
3
S0
LSP A
IS: 3 B
IS: 4 B
IS: 7 C
IS: 3 D
• Only best parallel adjacency is reported
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
40
P2P mode for back-to-back GE
Rtr-A
Rtr-A
Rtr-B
Rtr-B
Rtr-A
Rtr-B
interface fastethernet1/0
Pseudonode
isis network point-to-point
• No DIS election
• No CSNP transmission
• No Pseudo-node and extra link
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
41
Speeding up route installation
• Limit the # of leaves in the IGP
– only the BGP speakers are needed (
)
– rest: I-BGP
router isis
advertise passive-only
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
42
SPF, PRC and LSP-gen
Exponential BackOff
Timers
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
43
Backoff timer algorithm
• IS-IS throttles it main events
– SPF computation
– PRC computation
– LSP generation
• Throttling slows down convergence
• Not throttling can cause melt-downs
• The scope is to react fast to the first
events but, under constant churn, slow
down to avoid to collapse
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
44
Backoff timer algorithm
spf-interval <Max> [<Init> <Inc>]
• Maximum interval: Maximum amount of time the
router will wait between consecutives executions
• Initial delay: Time the router will wait before
starting execution
• Incremental interval: Time the router will wait
between consecutive execution. This timer is
variable and will increase until it reaches
Maximum-interval
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
45
spf-interval 10 100 1000
E1
Event1
E2
E3
SPF
100ms
E4
E5
E6
SPF
1000ms
E7
SPF
2000ms
4000ms
• Then 8000ms
• Then maxed at 10sec
• 20s without Trigger is required before resetting
the SPF timer to 100ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
46
Default Values
• Maximum-interval:
– SPF: 10 seconds
– PRC: 5 seconds
– LSP-Generation: 5
seconds
• Initial-wait:
– SPF: 5.5 seconds
• Incremental-interval:
– SPF: 5.5 seconds
– PRC: 5 seconds
– LSP-Generation: 5
seconds
– PRC: 2 seconds
– LSP-Generation: 50
milliseconds
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
47
Two-Way Connectivity Check
E
B
LSP
LSP
F
• For propagating Bad News,
1! LSP is enough
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
48
Timers for Fast Convergence
• Init Wait: 1ms
router isis
spf-interval 1 1 50
prc-interval 1 1 50
– 5.5 sec faster than default reaction!
– Optimized for the going down mode
• Exp Increment ~ S ms
• Max Wait ~ n * S ms
– CPU utilization < 1/n
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
49
Timer for Fast Convergence
router isis
lsp-gen-interval 5 1 50
• The timers are designed to optimize the
propagation of the information to other
nodes.
– Init-Wait = 1ms, 49ms faster than
default
– Exp-Inc = S, eg. 50ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
50
LSP Pacing and Flooding
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
53
LSP Pacing and Flooding
Int pos x/x
isis lsp-interval <>
• Pacing:
– Default: 33msecs inter-LSP gap
– backoff protection
– full database download
– suggest to keep the default
• Flooding
– flood/SPF trade-off
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
54
Link Protocol Properties
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
55
Link Protocol Properties
• Link Failure Detection
– the faster and more reliable, the better
• Dampening flapping links
– Fast signalling of a Down information
– Stable signalling of an UP information
– Freeze a flapping link in Down status
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
56
POS – Detection of a link failure
• Pos delay trigger line:
– hold time before reacting to a line alarm
– default is: immediate reaction
• Pos delay trigger path:
– hold time before reacting to a path alarm
– default is: no reaction
• Carrier-delay
– hold time between the end of the pos delay
holdtime and the bring down of the IOS interface
– default: 2000 msec
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
57
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
• Redundant for POS interfaces
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
58
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 60
pos delay triggers path 60
R1-ADM--PROTECTED_SONET_net--ADM-R2
• Should delay a little to allow for SONET
protection. Suggestion: 60msec
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
59
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 0
pos delay triggers path 0
R1-ADM--UNprotected_SONET_net--ADM-R2
• Should react as fast possible
– line default ok
– path default not ok
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
60
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 60
R1-DWDM--PROTECTED_DWDM_net--DWDM-R2
• Should delay for DWDM protection
– Suggestion: 60msec
– Alarm will be section or line
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
61
POS – Detection of a link failure
int pos 1/0
carrier-delay msec 8
pos delay triggers line 0
R1-DWDM--UNPROTECTED_DWDM_net--DWDM-R2
• Should react asap
– line: default ok
– path: not needed: default ok
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
62
POS – Bringing a down link back up
• Upon alarm clearance, POS Driver will wait
10seconds + <Carrier-Delay> before
turning the interface back up, hence
before triggering ISIS convergence
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
63
POS – Best for Convergence
• Very fast Link failure detection
– no need to tune the ISIS hello/holdtime
• Native anti-flap property of POS
– down info is signalled very fast
– up info is confirmed for 10s before
relaying to interface
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
64
Other types of Links
• Link Failure Detection
– If the native mode is too slow or if the link
has no failure detection capability
– ISIS Hello/Holdtime tuning
• Interface Dampening
– New feature to provide same Dampening
capability as BGP to the generic Interface
(applies to all types of interfaces)
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
65
Fast Hello’s
int serial0
isis hello-interval minimal
isis hello-multiplier 4
• Fast hello’s allow a dead timer of 1
second
• POS much faster/reliable
• Only useful when layer1/2 can’t help!
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
66
Operating this Design
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
67
ISIS Fast Convergence
Design
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
71
Design Tips
• POS as link type
int pos 1/0
carrier-delay msec 8
pos delay trigger …
–Do not tune ISIS hello’s and LSP-interval
• Design to minimize ISIS nodes,
links, prefixes
• Optimization: PRC, I-SPF, Flooding,
Parallel adjacencies, p2p GE
• SPF, PRC, LSP-Gen timers
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
router isis
spf-interval 1 1 50
prc-interval 1 1 50
lsp-gen-interval 5 1 50
72
Test Results
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
73
Test Scenari
• 12.0(19)S
• Carrier-delay configured to 8ms
• SPF, PRC, LSP-Gen Timers
– Default Timers
– Fast ISIS Configuration
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
74
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
1200 Nodes
4000 prefixes
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
75
A
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
10000pps
10000pps
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
B
76
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
ais
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
77
ISIS: 1200
Nodes, 4000
Leaves
BGP: 144000
prefixes
Ais
cleared
Agilent:
• A-B & B->A
• 10000 pps
• accuracy:
0.1ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
78
SPF Duration
1200
nodes
4000
prefixes
SPF duration
140
120
100
1
80
2
60
3
40
8
20
0
10-1
e5-1
e7-1
e3-1
f11-1
• SPF duration: ~ 100ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
79
Down
Link goes down
ms
6000.0
5000.0
4000.0
ab fast
ba fast
3000.0
ab def
ba def
2000.0
1000.0
0.0
1
Clarence Filsfils – Nanog 25
2
3
4
© 2002, Cisco Systems, Inc. All rights reserved.
5
6
7
8
9
10
80
Carrier-Delay
ISIS A->B 12.0(18)ST down event - AVG
3000
2500
msec
2000
1500
• Graph from Iain
0ms
8ms
12ms
16ms
50ms
2000ms
1000
500
0
1
carrier delay ms
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
81
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
82
Convergence
500
• 500 ISIS n
450
400
• 1000 ISIS p
350
300
• 80000 BGP p
250
200
• Accuracy:
0.1 ms
150
100
50
0
ISIS-NH A->B
ISIS-LNE A->B
Average [ms]
Clarence Filsfils – Nanog 25
BGP1 A->B
BGP2 A->B
Std-dev [ms]
© 2002, Cisco Systems, Inc. All rights reserved.
BGP3 A->B
• 10 iterations
83
Conclusion
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
84
Conclusion
• IGP convergence needs to be optimized
for Tight-SLA Services
• New development speed up convergence
without stability compromise
• Test results indicate that sub-second
convergence is realistic
• For sub-100ms Convergence, local action
based on precomputed tables might be
required
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
85
Agenda
• Introduction and SLA
• Sub-Second IGP Convergence
• Backbone Diffserv Design
• Conclusion
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
86
OverProvisioned
Backbone
DiffServ with a single class!
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
87
The Key is OverProvisioning
Offer must be higher than Demand
• The service that traffic receives is
dependent upon the ratio of traffic load to
available capacity
• More Bandwidth (offer) than traffic
(demand) means
– Low loss
– Low Latency
– Low Jitter
• Refs: [ROBERTS], [CHARNY], [BONALD]
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
88
Over-Provisioned Backbone
• A simple rule of design:
95-Percentile (5-min average Load) <= 50%
Link
which means
OverProvisioning (OP) > 2
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
89
Over-provisioning
(Source: Stephen Casner, Packet Design, NANOG 22)
Jitter Measurement Summary
for the Week
69 million packets transmitted
Zero packets lost
100% jitter < 700s
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
90
Drawback
• Risk related to provisioning failure
• Fate Sharing!
– No isolation between VPN, VoIP, Internet
• Expensive
– design for the aggregate!
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
91
Provisioning failure
• Capacity planning failures
– Small overprovisioning ratio: 2 vs 16
• Unexpected traffic demands
• Network failure situations
• Bandwidth unavailability
• Internet DoS Attack
FATE SHARING: Internet affects VoIP
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
92
“Not every week is like this”
(Source: Stephen Casner, Packet Design, NANOG 22)
99.99%
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
93
Recommendation: use DiffServ!
• Higher Availability of SLA
– Higher overprovisioning ratio (4 and
more)
– Service Isolation
• Cheaper
– Overprovisioning per Class!
• Mature Technology
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
94
Service Isolation
DSCP
ECN
• DiffServ Per-Hop Behavior
– Expedited Forwarding
>Low-latency/jitter scheduler (often a PQ)
– Assured Forwarding
>Bandwidth allocation and Multi-level
Congestion avoidance (RED)
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
95
Backbone Diffserv Design
• 2 or 3 Aggregate classes
• Edge DSCP marking policy to indicate class
Class
DSCP IP Prec
VoIP
40
5
Bus
32
4
Network
48
6
BE
0
0
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
Binary PHB
101 000 EF
100 000 AF1
110 000 AF1
000 000 Default
96
Backbone Diffserv Design
• VoIP
– EF PHB (a strict PQ)
– OP(V) = 4
• Business
– AF1 PHB: 90% of the remaining BW
– OP(V+B) = 2.25 = 1/0.45
• Internet
– AF2 PHB: 10% of the remaining BW
– OP(Aggr) = 1.25 = 1/0.8
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
97
Aggr Over-provisioning is
Expensive
• 1! DS Class:
– Agg <= 0.5
3C/1C
D
• 3 DS Classes
8
7
6
– V <= 0.25
5
4
– V+B <= 0.45
3
2
– Agg <= 0.8
1
0.44
0.41
0.38
0.35
0.32
0.29
0.26
0.23
0.2
0.17
0.14
0.11
0.08
0.05
0.02
V+B
0
Ex: V = 0.1, B =0.25
1 DS: D = 0.15
3 DS: D = 0.45
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
98
WRED Tuning
Link speed
P
Min. Th Max. Th.
OC3/STM-1
1292
194
1218
OC12/STM-4
5184
778
2826
OC48/STM-16 20000 3000
19384
• Based on simulations for at least 85%
utilisation with a mean queue size below
20msec
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
99
Typical Backbone Diffserv Design
class-map match-any VOIP
match ip precedence 5
class-map match-any BUS
match ip precedence 4
match ip precedence 6
!
policy-map OC3_POLICY
class VOIP
priority
class BUS
bandwidth percent remaining 90
random-detect prec 4 97 609 1
random-detect prec 6 97 609 1
class class-default
bandwidth percent remaining 10
random-detect prec 0 97 609 1
!
interface POS0/1
ip address 10.0.1.1 255.255.255.252
service-policy output OC3_POLICY
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
PE1
PE3
P1
P2
PE2
P3
P4
PE4
Static!
No inbound DiffServ Policy!
No marking, policing,
shaping in the core!
RED as congestion
avoidance for each Data
(TCP) Class
105
Provisioning is simple
• Same as ISIS, OSPF
• Configuration is done once and then it
remains static
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
106
Capacity Planning
• Aggregate Based
– DiffServ Isolation – risk hedging
• Per-Class Based
– OP per link/class
– Traffic Matrix per Class
– Better network utilization
• Significant edge qos deployment over last
24 months contribute to better NMS
support for QoS
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
107
Mature Technology
• EF: jitter due to non-EF
• AF: accuracy of BW allocation
• AF: latency as a function of AF load
• Even the rare cases are dealt with
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
108
EF: jitter charecteristic
voice packet latency on eng2 OC48
135000
130000
51v(200), 45bu,
150be
125000
ns
120000
30v(200), 45bu,
150be
115000
15v(200), 45bu,
150be
110000
105000
9v(200), 45bu,
150be
100000
95000
92
79
66
53
40
27
14
1
90000
percentile
E2b-OC48: Five times better than 500µs budget
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
109
EF: jitter characteristic
100
90
80
delay (µs)
70
v=30%, bus=45%,
be=150%
60
50
v=75%, bus=45%,
be=150%
40
30
20
10
0
1
E4-QOC48: 7 times better than 500us
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
110
AF: Bandwidth Allocation Accuracy
Accuracy of BUS bandwidth allocation
Measured Accuracy of BUS BW
[% of linerate]
0.15
0.1
0.05
0
-0.05
-0.1
-0.15
Expected BUS BW [% of linerate]
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
111
AF: Latency = f(load)
Max latency (us)
Latency in business class on OC48 with IMIX
1000000
100000
10000
1000
100
70
100
130
160
190
220
250
business load ratio
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
112
Optimised for even rare/cornercases
Input
Ports
Output
Tx side (tx-cos) of o/p line card Ports
Rx (rx-cos) side of i/p line card
E2: 2048 tofab VOQs
CEF
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
Crossbar Switch Fabric
16x16x8
E2: 128 fromfab Qs
16x8
113
Capacity Planning and
Monitoring
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
117
Capacity Planning and Monitoring
• A number of tools exist for capacity
planning:
–Per link statistics
–Core traffic matrices
–Active SLA monitoring
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
119
Link statistics
• packets and bytes through the class
• packets random-dropped
• packets forced-dropped
• no-buffer drops
• ignores
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
120
Core Traffic Matrix
• TMS: FIB accounting per non-recursive entry
• NetFlow v9 aggregated per BGP next-hop
• TE tunnel statistics (full-mesh req.)
• Reverse inference (research)
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
121
IPPM Infrastructure
POP2
POP3
POP1
POP4
SLA probes
P
PE
Ie. SAA
PE
PE
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
Shadow Router
123
MPLS-based
Technologies
TE, DS-TE, FRR
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
124
TE and SLA’s
• TE allows for the routing based on
constraints other than shortest-path
– bandwidth availability
– propagation latency
• DS-TE allows this for the aggregate and at
least one additional class-type
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
125
When TE is justified
• Drivers for MPLS TE deployment:
– Network asymmetry
– Unexpected demand
– Long bandwidth lead-times
• Drivers for DS-TE:
– above for aggregate; AND
– EF Load > 25% due to unoptimized
classic routing
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
126
MPLS FRR
• Link/Node Local Protection
– Pre-established and pre-computed
– Requires MPLS TE deployment
• When sub-second convergence is not
enough, but 50ms is required
Clarence Filsfils – Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
127
Tight-SLA IP Backbone
Conclusion
Clarence Filsfils - [email protected]
NANOG 25
© 2001, Cisco Systems, Inc. All rights reserved.
128
An overview of the Analysis
LLJ:Loss/Latency/Jitter
DiffServ: likely a Must
- EF(jitter) < 50us
- AF: 99.95% accuracy,
160us latency
Convergence
ISIS Sub-Second: Likely a Must
More Assurance, Cheaper
MPLS FRR: for <100ms
TE: if asymetric topology,
unexpected growth, long lead
times
DS-TE: if TE and EF
utilization per link risks to
be –too
high
Clarence Filsfils
Nanog 25
© 2002, Cisco Systems, Inc. All rights reserved.
129