Transcript Optical Interconnection Networks: The OSMOSIS Project

LEOS 2004
Optical Interconnection Networks:
The OSMOSIS Project
Ronald Luijten, Wolfgang E. Denzel
IBM Research, Zurich Research Laboratory, Rüschlikon, Switzerland
Richard R. Grzybowski, Roe Hemenway
Corning Incorporated, Science and Technology, Corning, NY, USA
OSMOSIS
High Performance Computing Systems (HPCS)
Illustrative HPCS example: IBM Blue Gene



2
Large distributed computing systems with 1000s of interconnected
processor/memory nodes
Proliferating processor performance and parallelism (Teraflops)
require interconnect performance to keep pace
Need for large-scale, high-bandwidth, low-latency packet switching
interconnection network
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
HPCS Interconnection Networks
X
X
X
X
X
X
X
X
X
X
X
X
Example of HPCS interconnection network
Illustrative HPCS example: IBM Blue Gene



3
Must be able to switch short data packets reliably at high sustained
rates with very low latency over arbitrary paths
Presently implemented as electronic packet switching networks,
but quickly approaching electronic limits with further scaling
Future could be based on maturing all-optical packet switching,
but need to solve the technical challenges and accelerate the cost
reduction of all-optical packet switching for HPCS
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Project
Optical Shared MemOry Supercomputer Interconnect System
 Sponsored by US Dept. of Energy, NNSA, as part of the
Accelerated Strategic Computing Initiative (ASCI)
 Partners: Corning and IBM
 Project duration: 2½ years (since Sep ‘03)
 Objective: solving the technical challenges and accelerating
the cost reduction of all-optical packet switches for HPCS
interconnects by
– building a full-function all-optical packet switch
demonstrator system
– showing the scalability, performance and cost paths
for a potential commercial system
4
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System - Overview
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
EQ
control
control
128
8
64
64
control links
central scheduler




5
(bipartite graph matching algorithm)
64-ports at 40 Gb/s port speed
Broadcast-and-select architecture
Combination of wavelength and space division multiplexing
fast switching based on SOAs
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System - Adapters
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
EQ
control
control
128
8
64
64
 Electronic Input Packet Buffers
• virtual output queues (VOQ)
 Optical Channel Transmitter
control links
• serialization & coding
• electro-absorption modulator
• DFB laser
central scheduler
(bipartite graph matching algorithm)
 Electronic control (Ingress)
• buffer control & scheduling
• control channel protocol
• FPGA-based
6
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System - Adapters
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
EQ
control
control
128
8
64
64
 Electronic Input Packet Buffers
• virtual output queues (VOQ)
 Optical Channel Transmitter
 2 Optical Channel Receivers
control links
• serialization & coding
• electro-absorption modulator
• DFB laser
 Electronic control (Ingress)
• buffer control & scheduling
• control channel protocol
• FPGA-based
7
• high-speed PIN photodiodes
• clock recovery
central scheduler
(optimized for fast clock acquisition)
(bipartite graph matching algorithm)
• error correction
 Electronic Output Packet Buffer
ingress & egress side physically collocated
The OSMOSIS Project (LEOS 2004)
• FIFO buffer
 Electronic control (Egress)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System – All-optical Switch
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
EQ
control
control
128
8
64
64
 8x1 WDM Multiplexer
• PLC AWG
control links
 Optical Amplifier
• EDFA with AGC
• >20dBm output power
• <7dB noise figure
central scheduler
(bipartite graph matching algorithm)
 1x128 Star Coupler
• 2-stage 1x8 / 1x16
• PLC splitters
8
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System – All-optical Switch
All-optical Switch
 Perfect Shuffle Interconnects
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
control
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
Fast SOA 1x8
Wavelength
Selector
Gates
control
1
2 Rx
 SOA 1x8 Fiber Selector Gates
128
SOA 1x8 Wavelength Selector Gates
8
64
 8x1 WDM Multiplexer
• PLC AWG
control links
 Optical Amplifier
• EDFA with AGC
• >20dBm output power
• <7dB noise figure
 1x128 Star Coupler
• 2-stage 1x8 / 1x16
• PLC splitters
EQ
EQ
control
64
• switching time <10 ns
• extinction ratio >20 dB (dynamic)
• gain >15 dB
• polarization-dependent gain 0.45 dB (typical)
• noise figure <6.5 dB
central
scheduler
• (bipartite
saturation
power
>17.5
dBm
graph
matching
algorithm)
• low wavelength-dependent power variation
 8x1 Combiner
• PLC combiner
 1x8 & 8x1 WDM Mux/Demux
• Dual PLC AWG
9
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System – Central Control
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
1
8x1
Tx
1x8
8x1
Star
Coupler
Optical
Amplifier
VOQs
2 Rx
1x128
1
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
EQ
control
control
128
8
64
64
control links
 Central Scheduler
• central clock and system synchronization
over control links
• central arbitration for entire fabric (FLPPR)
• control channel protocol for adapter
control links and switch command links
• FPGA-based
10
The OSMOSIS Project (LEOS 2004)
central scheduler
(bipartite graph matching algorithm)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Demonstrator System – Packet Scheduling
All-optical Switch
64 Ingress Adapters
64 Egress Adapters
8 Broadcast Units
128 Select Units
VOQs
Combiner
Tx
8x1
WDM
Mux
control
Optical
Amplifier
8x1
Tx
1x128
1
all-optical
packet transfer
5
1 packet
control
8
1x8
8x1
Star
Coupler
1
VOQs
2 Rx
64
waiting
1
Fast SOA 1x8
Fiber
Selector
Gates
EQ
control
1
Fast SOA 1x8
Wavelength
Selector
Gates
2 Rx
4b SOA
EQ
control
switch
command
128
64
control links
central scheduler
(bipartite graph matching algorithm)
2 request
11
The OSMOSIS Project (LEOS 2004)
3 central
arbitration
(FLPPR)
4a grant
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Delay/Throughput Measures
Delay [packet cycles]
with FIFOs
conventional
arbiter (PMM)

Virtual output queues (VOQ) in ingress
adapters avoid head-of-line blocking (i.e.
throughput saturation around 58%)

Novel Fast Low-latency Parallel Pipelined ARbitration (FLPPR) implementation
of heuristic iterative round-robin scheme
keeps low-load latency low, while still
allowing multiple iterations (k=6 for 64x64)

Dual receivers per output port improve
delay at high loads
(cost of >2 receivers not justified)
with VOQs
1 receiver
with FLPPR
2 receivers
k
1
0
50%
100%
Throughput
12
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Multistage Scalability
2048-port 2-level Fat Tree Topology
X
X
...
...
...
X
.. X
.. X
...
...
1
..
X
...
..
...
X
X
...
..
X
...
..
X
...
..
X
32 basic
OSMOSIS modules
of size 64x64
X
64 basic
OSMOSIS modules
of size 64x64
...
2048
 Single-stage scalability to 1000s of ports is not feasible economically
due to the square arbitration complexity and physical bulk of optical
components
 Scalability solution must be based on multistage topology involving
– electronic buffers between stages
– link-level flow control, routing and congestion control management
 Wavelength scalability in the basic modules allows further growth
without more stages
13
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Requirements Met
 Low switching overhead (<25%)
–
–
–
–
dead time for SOA switching
preamble for synchronization
packet header
forward error correction (FEC) bits
 Low bit-error rate (10-21), reliable delivery
– raw error rate target <10-10
– with single-error FEC on header and data →10-17
– with multiple-error detection and retransmission →10-21
 Low latency / high throughput
–
–
–
–
–
low optical-path delay, fast SOA switching
fast encoding/decoding → code block size compromise
fast central scheduling through pipelined implementation
two receivers per egress port
virtual output queues (VOQ)
 Scalability to 2048 nodes
– 3-stage, 2-level Fat Tree topology
14
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
OSMOSIS Project Status
 Physical and control architecture definition, simulations
25ns
 Initial lab demo of optical data path
Input waveform with intentional 3dB
path dependent power variation is
detected error free over three days
without benefit of FEC
 Progressing on plan to deliver final demonstrator system
 Engineering work required for commercially viable system,
aiming at 1/100 part count by integration of functional OE devices
to reduce cost, size and power consumption
15
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated
OSMOSIS
Acknowledgements
Other contributors:
 Michael Sauer, Martin Hu and Heath Rasmussen at Corning
 Ron Karfelt and the team at Photonic Controls, LLC
 Cyriel Minkenberg, François Abel, Raj Krishnamurthy, Ilias Iliadis,
Urs Bapst, Peter Mueller and Henry Brandt at IBM
 Marco Galli and the team at g&o embedded systems gmbh
16
The OSMOSIS Project (LEOS 2004)
© 2004 IBM Corporation & Corning Incorporated