Implementing 100 Gigabit Ethernet: A Practical Guide Joel Goergen
Download
Report
Transcript Implementing 100 Gigabit Ethernet: A Practical Guide Joel Goergen
Implementing 100 Gigabit
Ethernet: A Practical Guide
Joel Goergen
VP of Technology / Chief Scientist
1
Internet2 – JT February 2007
Special Note Regarding Forward
Looking Statements
This presentation contains forward-looking statements that involve substantial risks and uncertainties,
including but not limited to, statements relating to goals, plans, objectives and future events. All
statements, other than statements of historical facts, included in this presentation regarding our strategy,
future operations, future financial position, future revenues, projected costs, prospects and plans and
objectives of management are forward-looking statements. The words “anticipates,” “believes,”
“estimates,” “expects,” “intends,” “may,” “plans,” “projects,” “will,” “would” and similar expressions are
intended to identify forward-looking statements, although not all forward-looking statements contain these
identifying words. Examples of such statements include statements relating to products and product
features on our roadmap, the timing and commercial availability of such products and features, the
performance of such products and product features, statements concerning expectations for our products
and product features, and projections of revenue or other financial terms. These statements are based on
the current estimates and assumptions of management of Force10 as of the date hereof and are subject
to risks, uncertainties, changes in circumstances, assumptions and other factors that may cause the
actual results to be materially different from those reflected in our forward looking statements. We may
not actually achieve the plans, intentions or expectations disclosed in our forward-looking statements and
you should not place undue reliance on our forward-looking statements. In addition, our forward-looking
statements do not reflect the potential impact of any future acquisitions, mergers, dispositions, joint
ventures or investments we may make. We do not assume any obligation to update any forward-looking
statements.
2
Any information contained in our product roadmap is intended to outline our general product direction and
it should not be relied on in making purchasing decisions. The information on the roadmap is (i) for
information purposes only, (ii) may not be incorporated into any contract and (iii) does not constitute a
commitment, promise or legal obligation to deliver any material, code, or functionality. The development,
release and timing of any features or functionality described for our products remains at our sole
discretion.
Internet2 - JT February 2007
Per IEEE-SA Standards Board
Operations Manual, January 2005
At lectures, symposia, seminars, or
educational courses, an individual presenting
information on IEEE standards shall make it
clear that his or her views should be
considered the personal views of that
individual rather than the formal position,
explanation, or interpretation of the IEEE.
3
Internet2 - JT February 2007
Acronym Cheat Sheet
CFI – Call for Interest
DWDM – Dense Wavelength Division Multiplexing
EMI – Electro-magnetic Interference
Gbps – Gigabit per Second
HSSG – Higher Speed Study Group
ITU – International Telecommunications Union
IETF – Internet Engineering Task Force
JEDEC - Joint Electron Device Engineering Council
MAC – Media Access Control
MDI – Medium Dependent Interface
MSA – Multi Source Agreement
OIF – Optical Internetworking Forum
PCS – Physical Coding Sublayer
PMA – Physical Medium Attachment
PMD – Physical Medium Dependent
PHY – Physical Layer Device
SERDES – Serialize / De-serialize
SMF / MMF – Single Mode Fiber / Multi Mode Fiber
Tbps – Terabit per Second
WIS – WAN Interface Sublayer
XGMII – 10 Gigabit Media Indpendent Interface
4
Internet2 - JT February 2007
IEEE802.3 HSSG
My thoughts on where we are …..
5
Internet2 - JT February 2007
Current IEEE802.3 HSSG
Objectives as of January 2007
To date, the objectives within the study group
are:
– Support full duplex only
– Preserve the 802.3 / Ethernet frame format at the MAC
Client Service Interface
– Preserve the min and max FrameSize of current 802.3
standard
– Support a speed of 100Gbps at the MAC/PLS Interface
– Support at least 10km on SMF
– Support at least 100m on OM3 MMF
– Support a BER better then or equal to 10^-12 at the
MAC / PLS Service Interface
– Support at least 40km on SMF
6
Internet2 - JT February 2007
Upcoming IEEE802.3 HSSG March
Plenary
Looking for data to support the Broad Market
Potential of 100Gbps.
Looking for data to support the market for
40km on SMF.
7
Internet2 - JT February 2007
Shipping an IEEE802.3 HSSG
Vendor Compliant Product
Key architectural points
need to be addressed
from the MAC to the
PMD.
Likely to be mid 2009 or
later before the standard
is complete enough to
ensure compatibility
between vendors.
Higher Layers
LLC (Logical Link Control) or other MAC Client
MAC Control (Optional)
MAC
Reconciliation Sublayer (RS)
XGMII Optional Interface
64B/66B PCS
WIS
XGMII
XGXS
Optional
XGMII
Extender
XAUI
XGXS
XSBI
Optional
Interface
XGMII
64B/66B PCS
PMA
PMA
PMD
MAC = Media Access Control
MDI = Medium Dependent Interface
MDI
PCS = Physical Coding Sublayer
PMA = Physical Medium Attachment
MEDIUM
PMD = Physical Medium Dependent
WIS = WAN Interface Sublayer
10GBASE-W
XGMII = 10 Gigabit Media Independent Interface
8
Internet2 - JT February 2007
Not specified by
IEEE 802.3
XFI
PMD
MDI
MEDIUM
10GBASE-R
Developing a Standard
Ideas From Industry
Feasibility and Research
Industry
Pioneering
1 Year
Ad Hoc Efforts
Call for Interest
CFI July 18, 2006
Study Group
HSSG is here
Task Force
Q4 ‘07
Working Group Ballot
Sponsor Ballot
Force10 delivers 100G
Ethernet System
Standards Board Approval
’09 – ‘10
Publication
9
Internet2 - JT February 2007
100Gbps Architecture
Line Card
Switch Fabric
Back Plane
Power System
10
Internet2 - JT February 2007
Thoughts for a 100Gbps Line Card
CAM 200
MSPS
140
Fibre
100g
MAC
and
Phy
10 x
Inter
laken/
SPI-S
CEI11GSR
Inter
laken/
SPI-S
140
CAM 200
MSPS
Lookup
DataBase
SRAM
400 MHZ
DDRII+
50
72
Ingress
Packet
Parsing
25
Ingress
Lookup
50
36
Ingress
Packet
Edit
NPU
Ingress
Packet
Edit
Ingress
L ink List
SRAM
400 MHZ
QDRII+
Lookup
DataBase
SRAM 400
MHZ
DDRII+
Egress
Lookup
10 x
CEI11GSR
Inter
laken/
SPI-S
100
Clock, reset, PCI
Express, Test
Pins
11
Internet2 - JT February 2007
Inter
laken/
SPI-S
Ingress
Buffer
SDRAM
1 Ghz
DDR
72
123
256
32
10 x
16 x
CEI11GSR
Inter
laken/
SPI-S
CEI11GLR
50
TM
72
Egress
L ink List
SRAM
400 MHZ
QDRII+
123
256
Ingress
Buffer
SDRAM
1 Ghz
DDR
32
100
Clock, reset, PCI
Express, Test
Pins
Thoughts for a 100Gbps Line Card
600 Mhz x 192 bit datapath is barely feasible in 90 nm. Need to study
65nm feasibility and pick the right width and clock speed.
Memory components listed in the line card diagram are available now
or definitely before 2008.
32 x 10Gbps CEI serdes are possible in few 90 nm ASIC technologies.
Feasibility of Mac is well documented by belhadj_01_1106.pdf.
NPU Signal Pin Count 643 in this example including SERDES pins
Traffic Manager Signal Pin Count 1269 in this example including
SERDES pins. May have to be implemented using two chips to reduce
the pin count.
Die Size of the above chips are application dependent. For a
hardwired NPU,TM 65 nm may be the right process.
12
Internet2 - JT February 2007
Thoughts for a 100Gbps Front End
Optics – Short Reach (100m)
– 10 by 10Gbps
Optics – Long Reach (10km)
– 4 by 25Gbps
– 5 by 20Gbps
Optics – Extended Reach (40km)
– 4 by 25Gbps
– 5 by 20Gbps
13
Internet2 - JT February 2007
Thoughts for a 100Gbps Front End
CEI-11G
CEI-11G
CEI-11G
Receive
OPTIC
++20G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
++20G
Transmit
OPTIC
PMD
MUX /
SERDES
CEI-11G
CEI-11G
CEI-11G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
++20G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
“A” Possible Assembly
14
Internet2 - JT February 2007
MAC
SDD21 Magnitude (dB)
Thoughts for a 100Gbps Front End
0
-1
-2
-3
-4
-5
-6
-7
-8
-9
-10
-11
-12
-13
-14
-15
-16
-17
-18
-19
-20
0
XFI
• XFI based on 8 inches
+ 1 connector for fr-4
circuit boards
• Applications suggest 12
inches + 1 connector
for fr-4 circuit boards
• See proposed model
Note - XFI is only specified to 7 GHz.
Curve shown has been extrapolated
to 15 GHz using XFI equation
SDD21 (dB) = (-0.1- 0.78 *f ^(1/2) – 0.74* f)
1
2
3
4
5
6
7
8
9
10 11 12
Frequency (Hz)
15
Internet2 - JT February 2007
13 14
15
Thoughts for a 100Gbps
Switch Fabric
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
++20G
Transmit
MUX /
SERDES
++20G
++20G
++20G
Receive
MUX /
SERDES
++20G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
Data
Alignment
/ coding
Data
Coding
CEI-11G
CEI-11G
CEI-11G
CEI-11G
++20G
Transmit
MUX /
SERDES
++20G
++20G
++20G
Receive
MUX /
SERDES
++20G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
Digital
Crossbar
From Traffic
Manager 1
CEI-11G
To Traffic
Manager 1
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
++20G
Transmit
MUX /
SERDES
++20G
++20G
++20G
Receive
MUX /
SERDES
++20G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
Data
Alignment
/ coding
Data
Coding
CEI-11G
CEI-11G
CEI-11G
CEI-11G
++20G
Transmit
MUX /
SERDES
++20G
++20G
++20G
Receive
MUX /
SERDES
++20G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
CEI-11G
From Traffic
Manager N
16
To Traffic
Manager N
Switch Fabric ASIC
Back Plane
Switch Fabric
Internet2 - JT February 2007
CEI-11G
Back Plane
Thoughts for a 100Gbps
Switch Fabric
600 Mhz x 192 bit data path is barely feasible in 90 nm. Need to study
65nm feasibility and pick the right width and clock speed for a digital
cross bar, multiple ASIC implementation.
32 x 10Gbps CEI serdes are possible in few 90 nm ASIC technologies.
Need to study scaling 500 x 10Gbps SERDES across multiple ASICs.
Die Size of the above chips are application dependent. For a hardwired
switch fabric, 65 nm may be the right process.
Conceivable to see Master / Slave ASIC Architecture.
17
Internet2 - JT February 2007
Thoughts for a 100Gbps
Back Plane
Data Packet
Line Cards
--GbE / 10 GbE
RPMs
SFMs
Power Supplies
SERDES
Backplane
18
Traces
Mid Plane applications have complexities with holes and
connectors that make it incompatible for HSSG systems.
Internet2 - JT February 2007
Thoughts for a 100Gbps
Back Plane – Long Reach 30in.
0
Proposed CEI25 Channel
IEEE 802.3ap Informative Channel
SDD21 Magnitude (dB)
-10
-20
-30
-40
-50
-60
-70
0
SDD21 = -20*log10(e)*(b1*sqrt(f) + b2*f + b3*f^2 - b4*f^3)
b1 = 1.25e-5
b2 = 1.20e-10
b3 = 2.50e-20
b4 = 0.95e-30
f = 50Mhz to 15000Mhz
1
2
3
4
5
6
7
8
9
10 11 12 13 14 15
Frequency (Hz)
• Commercially available resin-based laminates exist to meet these
requirements (see OIF2006.097.00)
19
• Proposed CEI25 Channel Model (see OIF2006.047.00) – Chip
vendors would like to see 9dB better channel loss.
Internet2 - JT February 2007
Power System:
Thoughts for a 100Gbps Line Card
Optical
or Copper
Media
Forwarding
Engine
Media
Backplane
Reserved for Power
Network
Processor
S
E
R
D
E
S
S
E
R
D
E
S
20
Internet2 - JT February 2007
Architecture:
– Clean trace routing.
– Good power noise
control means better
than...
– Analog target
60mVpp ripple
– Digital target
150mVpp ripple
– Excellent SERDES to
connector signal flow to
minimize ground noise.
– Best choice for 100Gbps
systems.
Power System:
Thoughts for a 100Gbps Fabric
Reserved for Power
Digital
Cross Bar
S
E
R
D
E
S
S
E
R
D
E
S
21
Internet2 - JT February 2007
Architecture:
– Clean trace routing.
– Good power noise
control means better
than...
– Analog target
30mVpp ripple
– Digital target
100mVpp ripple
– Excellent SERDES to
connector signal flow to
minimize ground noise.
Power System:
300VDC to 500VDC System
PEM-A
DC PSU-A
F1
Backplane
RTNA
+
CB1
VDC
Vin ~
-VA
F2
Inrush
and soft
start
circuit
V+
Vo1
on/off
F3*
V-
RTN
-
V+
Vo2
on/off
F1
RTNB
+
F4*
CB2
V-
RTN
V+
Vo3
VDC
Vin ~
-VB
F3
-
DC PSU-B
on/off
PEM-B
F5*
V-
System Board
22
Internet2 - JT February 2007
RTN
Power System:
300VDC to 500VDC System
Current DC systems require large gauge wire. Reducing the current
capacity will allow smaller gauge wires.
Smaller gauge wires allow ease of use, yet still can carry the 10kw to
15kw required for a 100Gbps system.
Future AC systems may have to go to 480VAC to allow for smaller
gauge wires.
23
Internet2 - JT February 2007
Industry System Port Count Cycle
2002
GE
10 GE
2004
100’s Ports
10’s Ports
2006
2008
2010*
> 1000 Ports
100’s Ports
100 GE
> 100’s Ports
Standard In Development
24
Internet2 - JT February 2007
10’s Ports
Conclusions
Establish optical interfaces
Establish electrical interfaces, including circuit
board trace characteristics for both the front
end optics / electrical, and the back plane
electrical
Study the ASIC requirements for the Network
Processor Unit, the Traffic Manager, and the
Switch Fabric blocks.
Study the power subsystem.
– Cooling and EMI have to be a part of this.
25
Internet2 - JT February 2007
Thank You
26
Internet2 - JT February 2007
IEEE 802.3 HSSG
Reflector and Web
To subscribe to the HSSG reflector, send an email
to:
[email protected]
with the following in the body of the message:
subscribe stds-802-3-hssg <your first name> <your last
name>
end
SSG web page URL:
http://grouper.ieee.org/groups/802/3/hssg/index.html
John D’Ambrosia, Chair IEEE802.3 HSSG
– [email protected]
27
Internet2 - JT February 2007