Transcript NUMAの構成 - Keio University

Advanced direct networks and
asymmetric indirect networks
AMANO, Hideharu
Textbook pp.147 - 166
Networks for NORA machines/Clusters

Direct or Non-symmetric indirect networks



Nodes are connected with links.
Locality of communication can be used.
Extension to large size is easy.
Metrics of Direct interconnection network
（D and ｄ）

Diameter：D


degree: d


Number of hops between most distant two nodes
through the minimal path
The largest number of links per a node.
D represents performance and ｄ represents
cost
Recent trends:
Performance: Throughput
Cost: The number of long links
Other requirements





Uniformity：Every node/link has the same
configuration.
Extendability: The size can be easily
extended.
Fault Torelance: A single fault on link or
node does not cause a fatal damage on
the total network.
Embeddability: Emulating other networks
Bisection Bandwidth
bi-section bandwidth
The total amount of data
traffic between two halves of
the network.
Hypercube
0000
0100
1000
1100
0001
0101
1001
1101
0010
0110
1010
1110
0011
0111
1011
1111
ｋ-ａｒｙ ｎ-ｃｕｂｅ




Generalized mesh/torus
K-ary n digits number is assigned into each
node
For each dimension (digit), links are provided
to nodes whose number are the same except
the dimension in order.
Rap-around links (ｎ-1→０) form a torus,
otherwise mesh.
k-ary n-cube
００
０１
０２
3-ary 1-cube
１０
１１
１２
２０
２１
２２
3-ary 2-cube
k-ary n-cube
２ ００
０ ００
０１０
０ ２０
１ ００
１０１
００１
００ ２
１０
１１ ０
１ １１
０ １１
０１２
１２０
１２０
１２１
０ ２１
０ ２２
２０１
２０ ２
１０ ２
１１
２１２
3-ary 1-cube
１１２
２２１
２ ２２
3-ary 2-cube
１２２
3-ａｒｙ 3-ｃｕｂｅ
３-ary 4-cube
0***
1***
2***
Properties of k-ary n-cube




A class of networks which has Linear, Ring 2D/3-D mesh/torus and Hypercube（binary ncube） as its member.
1/n
Small d=2n but large D（O(k ))
Large number of neighboring links
k-ary n-cube has been a main stream of
NORA networks. Recently, small-n large-k
networks are trendy.
Interconnection networks for a large
scale parallel machines or clusters

Hypercubes → 2D/3D mesh/torus




small d, easy to implement
killer applications like partial differential equations
and image processing.
locality of communication
However, if the number of nodes is large, too
large diameter
Large scale interconnection networks
Advanced direct networks

Shuffle based networks


Extended mesh/torus



Midimew, RDT
Star Graph
Hierarchical networks


De Bruijn, Kautz, Pradhan
CCC, Hypernet
Circular networks

Circular Omega、MDCE
De Bruijn network
001
000
011
010
101
111
110
100
0
1
Routings for De Bruijn
001
000
011
010
101
111
110
100
0
1
Destination Routing
(001→101)
B(k，n)
..
..
0
..
1
..
k-1
K-ary n-digits
Characteristics of De Bruijn

Benefits

d=2k、D=n=logN
 When k=2, d=4、D=logＮ，that is, d of 2dimensional mesh but D of hypercube.
Problems





Optimal routing is difficult (not established yet).
Destination routing cannot make a best use of
communication locality.
No killer applications.
Self loop and duplicated links
Other shuffle based networks

Kautz


Pradhan



No self loops
Fault tolerant
Similar weak points of De Bruijn network
Irregular node numbers
Kautz network
210
The same number
should not be at the
neighboring digit
121
101
012
010
212
120
201
102
021
202
020
Pradhan network
100
1bit left rotation: shuffle +
Additional link
101
001
110
111
000
010
011
Circular networks

Circular Omega



Advantageous for one-way communication
Used in data-flow machine EM-4
MDCE(CCCB)


Hierarchical structure of Circular Omega
（Banyan）
Used in massively parallel machine RWC-1
Circular Omega network
000
001
000
001
010
011
010
011
100
101
100
101
110
111
110
111
Cube Connected Circular Banyan
Circular Banyan
3-Dimensional
Proposed for RWC-1
Star graph
ABCD
CBAD
DBCA
BACD
BACD
CABD
ACBD
CDAB
DCAB
CBDA
BDCA
ADCB
CDBA
DCBA
BDAC
DBAC
ADBC
DACB
ABDC
ACDB
CADB
BCDA
DABC
BADC
Connection n! nodes
Routing on Star graph
ABCD
CBAD
DBCA
BACD
BACD
CABD
ACBD
CDAB
DCAB
CBDA
BDCA
ADCB
CDBA
DCBA
BDAC
DBAC
ADBC
DACB
ABDC
ACDB
CADB
BCDA
DABC
BADC
If Ａ is top, change with arbitrary symbol, ＡＢＣＤ → ＤＡＢＣ
else, change with the symbol of destination
３（ｎ－１）／２
node
Hierarchical network

CCC(Cube Connected Cycles)


Hypernet



ｈｙｐｅｒｃｕｂｅ＋loop
Compete connection＋hypercube
Well combined, weak points of original
networks are vanished.
Complicated routing, gap between
hierarchies
CCC(Cube Connected
Cycles)
000
0
001
100
011
110
101
1
010
2
111
Hyper Net
h
i
b
c
b
d
o
a
j
c
e
d
f
e
f
g
g
k
h
l
m
a
p
Other links are used for
further upper hierarchy
n
Extended mesh/torus


Including mesh/torus structure
Extended links for performance enhancement



Reconfigurable Mesh
Midimew
RDT
RDT(Recursive Diagonal Torus)
Multicasting on the RDT
Asymmetric indirect networks


Intermediate position between direct and
indirect networks
High communication capability considering
cost

base-m n-cube（Hyper crossbar）


Fat Tree


SR2000、CP-PACS
CM-5，Some WS Clusters
Hyper-cross

ADENART
base-m n-cube
(Hyper crossbar)
crossbar
router
PU
Used in Toshiba’s Prodigy and Hitachi’s SR8000
HyperCross
(pi,pj)→ (pj,*),(*,pi)
0,0
0,3
Xbar
Xbar
Xbar
3,0
3,3
Used in ADENART by Matsushita
Fat Tree
Used in CM-5 and
PC Clusters( QsNet, Autonet )
Myrinet-Clos is actually a type of Fat-tree
Myrinet-Clos（１/2)

１２８nodes(Clos128)
Clos64+64
Myrinet-Clos(2/2)

512nodes
Summary



Recently, practical new topologies are not
proposed.
A lot of “made-in-Japan” networks
Asymmetric indirect networks will be widely
used.
Exercise

Calculate Diameter (D) and degree (d) of the
６-ａｒｙ 4-cube (mesh-type).