The International Grid Testbed: a 10 Gigabit Ethernet success story

in memoriam Bob Dobinson

Catalin Meirosu

on behalf of the IGT collaboration

GNEW 2004, Geneva

International Testbed

Contents

• www:

w

ho,

w

hy,

w

hat is IGT • Native Ethernet over lightpaths: from 1 Gbps to 10 Gbps • First demonstration of transatlantic native 10 GE over legacy OC-192 infrastructure • Future plans • Conclusions GNEW 2004 2

International Testbed

Who’s who in IGT

• • • • Universities and Research Institutes – Canada • Carleton University, Ottawa • • • • • • University of Alberta University of Victoria TRIUMF, Vancouver Université de Montreal McGill University, Montreal University of Toronto – The Netherlands • • University of Amsterdam – Switzerland CERN, Geneva Connectivity provided by CANARIE SURFnet ORION Netera Collaboration: CERN Openlab, the EU DataTAG project BCnet RISQ Sources of funding: IGT (Canarie Directed Research Grant), ESTA (IST-2001-33182) GNEW 2004 3

International Testbed

Current IGT topology

GNEW 2004 4

International Testbed

Motivation

• IGT is a research network for next generation

e-

science – Illustration of the application empowered network concept (see Bill St. Arnaud’s panel at SC2003) – Investigate emerging technologies in a demanding real life environment (see Wade Hong’s presentation at the CA*Net 4 Design Meeting May 26-27, 2003 ) – Real Time Farms for the ATLAS experiment at CERN (see Bryan Caron’s presentation at RISQ2003) GNEW 2004 5

International Testbed

The ATLAS Experiment at CERN

• Large collaboration of about 2000 scientists for an experiment operational from 2007 • Main goals: the discovery of new particles and exploring physics beyond the Standard Model • Challenging data collection and analysis systems – Online: 20 Gbps to the event filter – Offline: 4 TB/day to storage GNEW 2004 6

International Testbed

Ethernet: from local to global

• Ethernet, a brand name for LAN technology – Original Ethernet: shared media, half-duplex, distance limited by protocol – Modern Ethernet: point-to-point, full-duplex, switched, distance limited by the optical components • Cost effective !

GNEW 2004 7

International Testbed

Why native Ethernet long haul?

• More than 90% of the Internet traffic originates on an Ethernet LAN • Data traffic on the LAN increases due to new applications • Ethernet services with incremental bandwidth offer new business opportunities to carriers – See IEEE Communications Magazine, Vol. 42, No. 3, March 2004, on additional benefits for both the enterprise and the service providers • Why not native Ethernet ?

–

Scalability, reliability, service guarantees …

• All of the above are active research areas • Native Ethernet long haul connections can be used today as a

complement

to the routed networks,

not

as a

replacement

GNEW 2004 8

International Testbed

10 GE: a new Ethernet

• 10 GE – designed from the beginning for access to long haul networks – 40 km maximum distance specified by the standard … –

1550nm lasers

: optical amplifiers can be used to increase distance over dark fibre • State of the art: 250 km demonstrated in Denmark by the EU ESTA project [see Mikkel Olesen’s presentation at NORDunet2003] – what happens when you have to regenerate the signal ?

• no signal-agnostic regenerators deployed GNEW 2004 9

International Testbed

The 10 GE WAN PHY

• 10GE introduces a gateway from LAN to the WAN by means of the WAN PHY –

Compatible with existing WAN infrastructure

• Transmission rate • Encapsulation –

Partial use of the management bits of the SONET/SDH frame

• Today’s WAN PHY modules use SONET compliant optical components

Router OC192

traditional

LTE 3R LTE

OC192 Router

3R 3R LTE LTE WAN

WAN PHY

novel

WAN PHY 10GE switch/router 10GE switch/router

GNEW 2004 10

International Testbed

Demo during ITU Telecom World '03

Intel Itanium-2 Intel Xeon Ixia 400T 10GE WAN PHY 10GE LAN PHY OC192c Force10 E 600 Ottawa Cisco ONS 15454 Cisco ONS 15454 Cisco ONS 15454 Toronto Chicago Cisco ONS 15454 Amsterdam Cisco ONS 15454 Force10 E 600 Geneva

10 GE WAN PHY over an OC-192c circuit using lightpaths provided by SURFnet and CANARIE 9.24 Gbps using traffic generators 6 Gbps using UDP on PCs GNEW 2004 5.65 Gbps using TCP on PCs 11

HP Itanium-2 HP Itanium-2 Ixia 400T

International Testbed

Results on the transatlantic 10 GE

Single stream UDP throughput Single stream TCP throughput •Data rates are limited by the PC, even for our memory-to-memory tests •UDP uses less resources than TCP on high bandwidth-delay product networks GNEW 2004 12

International Testbed

WAN PHY over DWDM

HP Itanium-2 10GE LAN Intel Xeon Ixia 400T Force10 E600 10 GE WAN DWDM DWDM 10 GE WAN Force10 E600 Ixia 400T 10GE LAN HP Itanium 2 Amsterdam Geneva HP Itanium 2

• Direct lambda access from the provider is required • The DWDM transceiver card as “LTE” GNEW 2004 13

International Testbed

What next ?

• More 10 Gigabit Ethernet experiments – Extend the reach of our WAN PHY connection • Adds more latency into the testbed, hence increased pressure on protocols • Might allow us to identify technical showstoppers – Disk-to-disk transfers – Comparative study of data transfer protocols for 10 Gbps networks GNEW 2004 14

International Testbed

What next ?

• Support for real-time access to remote computer farms during the ATLAS testbeam run in 2004 – 1 Gbps connectivity is sufficient as a proof of concept – Sites in Copenhagen, Cracow and Edmonton will process in real time data acquired at the CERN testbeam • This is part of the feasibility study for remote real time processing in ATLAS GNEW 2004 15

International Testbed

Conclusions

• IGT demonstrated native 10 Gigabit Ethernet over lightpaths • 10GE WAN PHY is the technology that enables inter-continental native Ethernet • IGT will support remote real time applications, with emphasis on data collection and analysis GNEW 2004 16

International Testbed

References

• Bill St. Arnaud – contribution to the “Strategies for Application Empowered Networks” panel at SC2003 • Bryan Caron – “The GARDEN Project: A Testbed for High Bandwidth Real-Time Applications”, http://www.risq.qc.ca/risq2003 canw2003/ppt/23PM_Bryan_Caron.pdf

• Wade Hong – “CA*net 4 International Grid Testbed ”, http://lightpath.physics.carleton.ca/C4DesignIGTMay27.ppt

• Mikkel Olesen – “The use of dark fibre in Forskningsnttet”, the 21 st NORUnet Conference, http://www.nordunet2003.is/smasidur/presentations/MikkelOlesen.ppt

GNEW 2004 17