Transcript Third Generation Routers

How scalable is the capacity of
(electronic) IP routers?
High Performance
Switching and Routing
Telecom Center Workshop: Sept 4, 1997.
Nick McKeown
Professor of Electrical Engineering
and Computer Science, Stanford University
[email protected]
http://www.stanford.edu/~nickm
1
Why ask the question?
Widely held assumption:
Electronic IP routers will not keep up with
link capacity.
Background:
Router Capacity = (number of lines) x (line-rate)
•
•
•
•
Biggest router capacity 4 years ago ~= 10Gb/s
Biggest router capacity 2 years ago ~= 40Gb/s
Biggest router capacity today ~= 160Gb/s
Next couple of generations: ~1-40Tb/s
2
Why it’s hard for capacity to
keep up with link rates
Packet processing
power
Link rate
3
Why it’s hard for capacity to
keep up with link rates
Packet processing Power
Link Speed
1000
10000
2x / 2 years
1000
2x / 7 months
100
100
10
10
1
1
1985
1990
1995
2000
1985
1990
1995
Fiber Capacity (Gbit/s)
Spec95Int CPU results
10000
2000
0,1
0,1
TDM
DWDM
Source: SPEC95Int & David Miller, Stanford.
4
Instructions per packet
What will happen
Instructions
per packet
What we’d like: (more features)
QoS, Multicast, Security, …
time
5
What limits a router’s capacity?
Limited by • It’s a packet switch:
memory
– Must be able to buffer every packet
random
for an unpredictable amount of time.
access time
Limited by
memory
random
access time
• Hop-by-hop routing:
– Once per ~1000bits it must index into
a forwarding table with ~100k entries.
• [Optional QoS support
– Very complex per-packet processing]
6
What really limits the capacity?
• At first glance: the random access
time to memory.
• In fact, this can be solved by more
parallelism (replication and pipelining).
• Dilemma: But parallelism requires
more power and space.
7
What really limits the capacity?
Suggestion:
– Don’t assume optics will oust CMOS in IP
routers because of increased system
capacity.
– It might oust CMOS because of reduced
(power x space) for a given capacity.
8
Outline
• A brief history of IP routers
• Where they will go next
– Incorporating optics into routers
– More parallelism (with or without optics)
9
First Generation Routers
Shared Backplane
CPU
Route
Table
Buffer
Memory
Line
Interface
Line
Interface
Line
Interface
MAC
MAC
MAC
Typically <0.5Gb/s aggregate capacity
10
First Generation Routers
Queueing Structure: Centralized Shared Memory
Large, single dynamically
allocated memory buffer:
N writes per “cell” time
N reads per “cell” time.
Limited by memory
bandwidth.
Input 1
Input 2
Input N
Numerous work has proven
and made possible QoS:
–
–
–
–
–
Memory
Controller
Fairness
Delay Guarantees
Delay Variation Control
Loss Guarantees
Statistical Guarantees
Output 1
Output 2
Output N
11
Second Generation Routers
CPU
Route
Table
Buffer
Memory
Line
Card
Line
Card
Line
Card
Buffer
Memory
Buffer
Memory
Buffer
Memory
Fwding
Cache
Fwding
Cache
Fwding
Cache
MAC
MAC
MAC
Typically <5Gb/s aggregate capacity
12
Second Generation Routers
Queueing Structure: Combined Input and Output
Queueing
1 write per “cell” time
Rate of writes/reads
determined by bus speed
1 read per “cell” time
Bus
13
Third Generation Routers
Switched Backplane
Line
Card
CPU
Card
Line
Card
Local
Buffer
Memory
Routing
Table
Local
Buffer
Memory
Fwding
Table
Fwding
Table
MAC
MAC
Typically <50Gb/s aggregate capacity
14
Third Generation Routers
Queueing Structure
1 write per “cell” time
Rate of writes/reads
determined by switch
fabric speedup
1 read per “cell” time
Switch
Arbiter
Flow-control
backpressure
Per-flow/class or peroutput queues (VOQs)
Per-flow/class or perinput queues
15
Third Generation Routers
7’
•
Size-constrained: 19” or 23” wide.
•
Power-constrained.
19” or 23”
16
Complex linecards
Typical IP Router Linecard
Optics
Physical
Layer
Framing
&
Maintenance
Lookup
Tables
Buffer
& State
Memory
Packet
Processing
Buffer Mgmt
&
Scheduling
Buffer Mgmt
&
Scheduling
Buffer
& State
Memory
Switch
Fabric
Arbitration
OC192c linecard:
~10-30M gates
~2Gbits of memory
~2 square feet
>$10k cost
100’s of Watts
“Backplane”
17
Fourth Generation Routers/Switches
Optics inside a router for the first time
The LCS
Protocol
Optical links
1000’s
of feet
Switch Core
Linecards
0.3 - 10Tb/s routers in development
18
Where next?
• Incorporating (more) optics into a
router.
• More parallelism (with or without
optics).
19
Incorporating optics into a router
• Replacing the switch fabric with an
optical datapath.
• Increasing the internal “cell” size to
reduce rate of arbitration and
reconfiguration.
20
Replacing the switch fabric with optics
Typical IP Router Linecard
Lookup
Tables
Packet
Processing
Optics
Physical
Layer
Typical IP Router Linecard
Buffer
& State
Memory
electrical
Buffer Mgmt
&
Scheduling
Switch
Fabric
Framing
&
Maintenance
Buffer Mgmt
&
Scheduling
Buffer
& State
Memory
Packet
Processing
Optics
Physical
Layer
Lookup
Tables
Buffer Mgmt
&
Scheduling
Packet
Processing
Optics
Framing
&
Maintenance
Buffer Mgmt
&
Scheduling
Buffer
& State
Memory
1.
MEMs
2.
Fast tunable lasers + passive optical couplers
Buffer Mgmt
&
Scheduling
Switch
Fabric
Framing
&
Maintenance
Buffer
& State
Memory
Typical IP Router Linecard
optical
Buffer
& State
Memory
Buffer Mgmt
&
Scheduling
Physical
Layer
Arbitration
Candidate
technologies
Typical IP Router Linecard
Lookup
Tables
Buffer
& State
Memory
Req/Grant
Arbitration
Buffer
& State
Memory
Lookup
Tables
Buffer Mgmt
&
Scheduling
Packet
Processing
Optics
Framing
&
Maintenance
Req/Grant
Physical
Layer
Buffer Mgmt
&
Scheduling
Buffer
& State
Memory
21
Replacing the switch fabric with
optics
• Most common internal “cell” size is 64
bytes (50ns @ OC192, 12ns @ OC768)
• Too fast for arbitration
• Too fast for reconfiguration
• What we’ll see:
– Increased cell length
– E.g. switch bursts of cells
– But less efficient.
22
More parallelism
• Parallel packet buffers
• Parallel lookup tables
• Multiple parallel routers
23
Multiple parallel routers
What we’d like:
IP Router capacity
100s of Tb/s
R
R
NxN
R
R
The building blocks we’d like to use:
R
R
R
R
24
Why this might be a good idea
•
•
•
•
Larger overall capacity
Faster line rates
Redundancy
Familiarity
– “After all, this is how the Internet is
built”
25
Multiple parallel routers
Load Balancing architectures
R
R
R
R
R
R
R/k
1
2
…
…
k
R/k
R
R
R
26
Method #1: Random packet loadbalancing
Method: As packets arrive they are randomly
distributed, packet by packet over each router.
Advantages:
– Almost unlimited capacity
– Load-balancer is simple
– Load-balancer needs no packet buffering
Disadvantages:
– Random fluctuations in traffic a each router is loaded
differently
• Packets within a flow may become mis-sequenced
• It is not possible to predict the system performance
27
Method #2: Random flow loadbalancing
Method: Each new flow (e.g. TCP connection) is
randomly assigned to a router. All packets in a
flow follow the same path.
Advantages:
–
–
–
–
Almost unlimited capacity
Load-balancer is simple (e.g. hashing of flow ID).
Load-balancer needs no packet buffering.
No mis-sequencing of packets within a flow.
Disadvantages:
– Random fluctuations in traffic a each router is loaded
differently
• It is not possible to predict the system performance
28
Observations
• Random load-balancing: It’s hard to predict
system performance.
• Flow-by-flow load-balancing: Worst-case
performance is very poor.
If designers, system builders, network
operators etc. need to know the
worst case performance, random
load-balancing will not suffice.
29
Method #3: Intelligent packet
load-balancing
Goal: Each new packet is carefully
assigned to a router so that:
• Packets are not mis-sequenced.
• The throughput is maximized and
understood.
• Delay of each packet can be controlled.
We call this “Parallel Packet Switching”
30
Method #3: Intelligent packet loadbalancing
Parallel Packet Switching
Router
R/k
rate, R
1
R/k
1
1
rate, R
2
rate, R
N
N
rate, R
k
Bufferless
31
Parallel Packet Switching
• Advantages
–
–
–
–
–
Single-stage of buffering
No excess link capacity
kh a power per subsystem i
kh a memory bandwidth i
kh a lookup rate i
32
Precise Emulation of a Shared
Memory Switch
Shared Memory Switch
1
N
=?
N
N
Parallel Packet Switch
1
1
N
N
33
Parallel Packet Switch
Theorem
1. If S > 2k/(k+2) @ 2 then a parallel
packet switch can precisely emulate
a FCFS shared memory switch for
all traffic.
34
Example of an IP Router with Parallel
Packet Switching
10Tb/s router
R/k
R/k
160Gb/s
1
1
1
rate, R
2
160Gb/s
1024
1024
rate, R
16
Overall capacity 160Tb/s
35
My conclusions
• The capacity of electronic IP routers
will scale a long way yet.
• The opportunity of optics is to reduce
power and space
– By using optics within the router.
– By replacing routers with circuit
switches.
36