Transcript Dynamic Networks

Interconnection Networks
Shanghai Jiao Tong University 2012
Indirect Networks or
Dynamic Networks
Guihai Chen
…with major presentation contribution from José Flich, UPV
(and Cell BE EIB slides by Tom Ainsworth, USC)
Questions in mind
Difference between Static/direct and Dynamic/indirect Networks
Shanghai Jiao Tong University 2012
Why Multi-Stage Interconnection Networks
Large Switch and Small Switch
How to design non-blocking MINs
2
Outline
Network Topology
Shanghai Jiao Tong University 2012
Preliminaries and Evolution
Centralized Switched (Indirect) Networks
From non-blocking crossbar to blocking MINs
From blocking MINs to non-blocking MINs
3
Network Topology
Preliminaries and Evolution
Shanghai Jiao Tong University 2012
• One switch suffices to connect a small number of devices
– Number of switch ports limited by VLSI technology, power
consumption, packaging, and other such cost constraints
• A fabric of interconnected switches (i.e., switch fabric or network
fabric) is needed when the number of devices is much larger
– The topology must make a path(s) available for every pair of
devices—property of connectedness or full access (What paths?)
• Topology defines the connection structure across all components
– Bisection bandwidth: the minimum bandwidth of all links crossing
a network split into two roughly equal halves
– Full bisection bandwidth:
› Network BWBisection = Injection (or Reception) BWBisection= N/2
– Bisection bandwidth mainly affects performance
• Topology is constrained primarily by local chip/board pin-outs;
secondarily, (if at all) by global bisection bandwidth
4
Network Topology
Preliminaries and Evolution
Shanghai Jiao Tong University 2012
• Several tens of topologies proposed, but less than a dozen used
• 1970s and 1980s
– Topologies were proposed to reduce hop count
• 1990s
– Pipelined transmission and switching techniques
– Packet latency became decoupled from hop count
• 2000s
– Topology still important (especially OCNs, SANs, P2P Overlays,
DCNs) when N is high
– Topology impacts performance and has a major impact on cost
5
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Crossbar network
– Crosspoint switch complexity increases quadratically with the
number of crossbar input/output ports, N, i.e., grows as O(N2)
– Has the property of being non-blocking
0
1
2
3
4
5
6
0
7
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
1
2
3
4
5
6
7
6
Network Topology
Shanghai Jiao Tong University 2012
From Crossbar to MINs
7
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Multistage interconnection networks (MINs)
– Crossbar split into several stages consisting of smaller crossbars
– Complexity grows as O(N × log N), where N is # of end nodes
– Inter-stage connections represented by a set of permutation
functions
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
Omega topology, perfect-shuffle exchange
8
Network Topology
Appendix
Shanghai Jiao Tong University 2012
• Shuffle function
– N= 0…N-1
– f(i)= 2i, when I <N/2
– f(i)=(2i+1), mod N when i ≥ N/2
– Often used as a connection pattern
– unshuffle function, also often used
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
perfect-shuffle
unshuffle
9
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0000
0001
0000
0001
0010
0011
0010
0011
0100
0101
0100
0101
0110
0111
0110
0111
1000
1001
1000
1001
1010
1011
1010
1011
1100
1101
1100
1101
1110
1111
1110
1111
16 port, 4 stage Omega network
10
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0000
0001
0000
0001
0010
0011
0010
0011
0100
0101
0100
0101
0110
0111
0110
0111
1000
1001
1000
1001
1010
1011
1010
1011
1100
1101
1100
1101
1110
1111
1110
1111
16 port, 4 stage Baseline network
11
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0000
0001
0000
0001
0010
0011
0010
0011
0100
0101
0100
0101
0110
0111
0110
0111
1000
1001
1000
1001
1010
1011
1010
1011
1100
1101
1100
1101
1110
1111
1110
1111
16 port, 4 stage Butterfly network
12
Network Topology-Correction to Butterfly
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0000
0001
0000
0001
0010
0011
0010
0011
0100
0101
0100
0101
0110
0111
0110
0111
1000
1001
1000
1001
1010
1011
1010
1011
1100
1101
1100
1101
1110
1111
1110
1111
16 port, 4 stage Butterfly network
13
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0000
0001
0000
0001
0010
0011
0010
0011
0100
0101
0100
0101
0110
0111
0110
0111
1000
1001
1000
1001
1010
1011
1010
1011
1100
1101
1100
1101
1110
1111
1110
1111
16 port, 4 stage Cube network
14
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Multistage interconnection networks (MINs)
– MINs interconnect N input/output ports using k x k switches
› logkN switch stages, each with N/k switches
› N/k(logkN) total number of switches
– Example: Compute the switch and link costs of interconnecting
4096 nodes using a crossbar relative to a MIN, assuming that
switch cost grows quadratically with the number of input/output
ports (k). Consider the following values of k:
› MIN with 2 x 2 switches
› MIN with 4 x 4 switches
› MIN with 16 x 16 switches
15
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Example: compute the relative switch and link costs, N = 4096
cost(crossbar)switches = 40962
cost(crossbar)links = 8192
relative_cost(2 × 2)switches = 40962 / (22 × 4096/2 × log2 4096) = 170
relative_cost(2 × 2)links = 8192 / (4096 × (log2 4096 + 1)) = 2/13 = 0.1538
relative_cost(4 × 4)switches = 40962 / (42 × 4096/4 × log4 4096) = 170
relative_cost(4 × 4)links = 8192 / (4096 × (log4 4096 + 1)) = 2/7 = 0.2857
relative_cost(16 × 16)switches = 40962 / (162 × 4096/16 × log16 4096) = 85
relative_cost(16 × 16)links = 8192 / (4096 × (log16 4096 + 1)) = 2/4 = 0.5
16
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Relative switch and link costs for various values of k and N
(crossbar relative to a MIN)
0-0.5
350
300
250
200
150
100
50
0
0.5-1
1-1.5
1.5-2
2
1.5
1
8192
512
32
N
2
2
16
128
1024
8192
k
Relative switch cost
8192
512
0.5
32
0
N
2
2
16
128
1024
8192
k
Relative link cost
17
Network Topology
Shanghai Jiao Tong University 2012
From blocking to non-blocking again
18
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Reduction in MIN switch cost comes at the price of performance
– Network has the property of being blocking
– Contention is more likely to occur on network links
› Paths from different sources to different destinations share one or
more links
0
1
2
3
4
5
6
7
0
0
1
2
3
4
5
6
7
non-blocking topology
X
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
blocking topology
19
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• How to reduce blocking in MINs? Provide alternative paths!
– Use larger switches (can equate to using more switches)
› Clos network: minimally three stages (non-blocking)
» A larger switch in the middle of two other switch stages
provides enough alternative paths to avoid all conflicts
– Use more switches
› Add logkN - 1 stages, mirroring the original topology
» Rearrangeably non-blocking
» Allows for non-conflicting paths
» Doubles network hop count (distance), d
» Centralized control can rearrange established paths
› Benes topology: 2(log2N) - 1 stages (rearrangeably non-blocking)
» Recursively applies the three-stage Clos network concept to
the middle-stage set of switches to reduce all switches to 2 x 2
20
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16 port Crossbar network
21
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16 port, 3-stage Clos network
22
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16 port, 5-stage Clos network
23
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
16 port, 7 stage Clos network = Benes topology
24
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
Alternative paths from 0 to 1. 16 port, 7 stage Clos network = Benes topology
25
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
Alternative paths from 4 to 0. 16 port, 7 stage Clos network = Benes topology
26
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
0
0
1
1
2
2
3
3
4
4
5
5
6
6
7
7
8
8
9
9
10
10
11
11
12
12
13
13
14
14
15
15
Contention free, paths 0 to 1 and 4 to 1. 16 port, 7 stage Clos network = Benes topology
27
Network Topology
Centralized Switched (Indirect) Networks
Shanghai Jiao Tong University 2012
• Bidirectional MINs
• Increase modularity
• Reduce hop count, d
• Fat tree network
Network
Bisection
0
1
2
3
– Nodes at tree leaves
– Switches at tree
vertices
– Total link bandwidth
is constant across all
tree levels, with full
bisection bandwidth
– Equivalent to folded
Benes topology
– Preferred topology in
many SANs
4
5
6
7
8
9
10
11
12
13
14
15
Folded Clos = Folded Benes = Fat tree network
28
Network Topology
Shanghai Jiao Tong University 2012
Myrinet-2000 Clos Network for 128 Hosts
• Backplane of the M3E128 Switch
• M3-SW16-8F fiber line
card (8 ports)
http://myri.com
29
Network Topology
Shanghai Jiao Tong University 2012
Myrinet-2000 Clos Network for 128 Hosts
• “Network in a Box”
• 16 fiber line cards
connected to the
M3-E128 Switch
backplane
http://myri.com
30
Network Topology
Myrinet-2000 Clos Network Extended to 512 Hosts
Shanghai Jiao Tong University 2012
http://myri.com
31
Assignment 2-2
Shanghai Jiao Tong University 2012
Chose one of the following exercises:
• Calculate how many permutations nxn Omega
network could support.
• Prove that folded Clos network, folded Bens
network, and Fat tree network are isomorphic to each
other.
Parallel Processing, Low-Diameter Architectures
Slide 32
SJTU@Fall 2012