Transcript UltraNet Engineering

The DOE UltraScience Network
(aka UltraNet)
GNEW2004
March 15, 2004
Wing/Rao
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Talk Outline





What are we trying to do?
Why do it?
How (we all sort of know don’t we)?
How can we get there from here?
…Where will that be?
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
DOE Science
 DOE has bigger problems (drivers) …
(and less money) than any other threeletter agency
 But with it, we are trying to do things like:
 Climate Modeling
 Supernova modeling
 Genomics (and proteomics)
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Climate Modeling
North American model at 400Km
resolution
North American model at 60Km
resolution
The Rockies block the atmospheric circulation and are important
geneses regions for the weather systems that form the North Atlantic
Storm Track. They are poorly represented in low resolution models.
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Type II CoreCollapse Supernova
QuickTime™ and a
Video decompressor
are needed to see this picture.
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Computational Molecular Biology
.25TF
Membrane with possible uses for bioremediation
11TF
400 TF
>1.0 PF
Molecular dynamics of a
lipopolysaccharide (LPS)
Nanosecond molecular
dynamics of the LPS
membrane of
Pseudomonas aeruginosa
QM/MM molecular
dynamics of
membrane plus
mineral
Energy transduction across
membranes is still out of
reach at 1.0PF
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Billions of Bases in GenBank
According to the GOLD database,
there are 146 published genomes,
344 prokaryotic ongoing genomes
projects, and 243 eukaryotic
ongoing genome projects.
1982
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
1986
1990
1994
1998
2002
Growth of Proteomic Data vs. Sequence
Data
1000
100
PetaBytes
10
1
0.1
0.01
0.001
Proteomic data
0.0001
GenBank
19
88
19
90
19
92
19
94
19
96
19
98
20
00
20
02
20
04
20
06
20
08
20
10
20
12
20
14
20
16
0.00001
Years
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
PetaScale Science
 Climate modeling today => 50-100 TB data sets
 Still adding carbon chemistry
 Still refining resolution
 You saw DOE Road Map table showing PB
requirements in 2-3 years
 Supernova modeling just going from 2 1/2
dimensions to 3D - this is a PB problem today
 Genomics and Proteomics will dwarf these
 …and then of course there is High Energy
…but
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
We run hard into network limits…
 There is some VERY nice work being done on TCP
 But that will just emphasize router delay
 And expose congestion, even where it coexists with it
 There is nice work being done to extend MPLS
 But that does not eliminate jitter
 And imposes additional overhead
 And there is still the tyranny of layers
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Jitter - ORNL / Atlanta
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Science Applications - Bit Rate Budget
End-to-end
Bit-Rate
100
Gbps
DWDM
(One
Lambda)
Research Opportunities
40
Gbps
OC-768
10
Gbps
OC192
ESnet
ESnet
5.0
Gbps
2.5
Gbps
SONET
(One Channel)
(One Flow)
Site
Security
Cluster
Firewall
OC-48
1.0
Gbps
.65
Gbps
Apps
TCP/IP
ESnet
SITE
LAN
Optical Backbone
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
IP Network Services
Host
ESnet
Host
End-to-end
Performance
Gap
GigE
One Host
What are we trying to do?
 We are trying to break out of the limitations of
conventional networks
 TCP/IP
 Firewalls, and layers
 Small-frame technologies
 Take advantage of the real throughput available in
optical networks today at the SONET layer
 Develop the techniques needed within 2-3 years
for DOE Peta-scale science
 Use switched-circuits as a solution we can start
on today
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Restating the Problem  Networks will soon offer 40 Gbs channels
 Typical multi-stream throughput limited to
5-6Gbs, and programming multi-steam is
hard - hero efforts yield 9-10Gbs
 Typical single-steam throughput to a
single application ~1Gbs, even on a tuned
network - hero effort yields 5-6Gbs
 Firewalls throw it away anyway
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Solution  We won’t have line-rate optical packet-switching anytime
soon
 No photonic transistors
 No photonic memory buffers
 We can do optical circuit switching
 Useless for most commercial networks
 Doesn’t aggregate for economy of scale
 But bulk P2P purchases can be extremely attractive
 A good match for DOE high-performance science
applications which require both bandwidth and precision
 Petabyte file transmission in 2-3 years
 Tera-scale interactive visualization ASAP
 Remote real-time instrument control now
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
UltraNet Research Testbed
 Build a sparse, lambda-switching, dedicated
channel-provisioning testbed
 Connect hubs close to DOE’s largest Science
users (but let the user labs pay last-mile costs)
 Provide an evolving matrix of switching
capabilities
 Separately fund research projects (e.g., highperformance protocols, control, visualization)
that will exercise the network and directly support
applications at the host institutions
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
The Resources:
 Off hours capacity from Esnet
 Probably 2 x OC48 between Sunnyvale and
Chicago in the northern route
 Dedicated Lambdas on NLR
 Two 10G lambdas between Chicago and
Sunnyvale
 Possibly more in year two or three
 Two dedicated lambdas on the ORNL
Chicago-Atlanta connector
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Resources Cont.
 A progression of switching technologies
 Start with Ciena, Cisco, or Sycamore
 Migrate to Calient, Glimmerglass all-optical or a hybrid
 Some Local Storage
 Logistical Storage Depot (prior to local application
development)
 Progression of experimental point-to-point
transport technologies (plus legacy)
 Fiber channel
 Infiniband
 Migrate to the Production ESnet environment
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
CERN
Science UltraNet Phase I
Seattle
PNNL
2 x10 Gbps
LBL
StarLight
NERSC
BNL
2 x10 Gbps
SLAC
Sunnyvale
ANL
FNAL
CalTech
ORNL
JLab
SOX
Research Exchange Point
DOE University Partners
DOE National Lab
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Access via production networks
Research 10 Gbps lambda
The Physical View From 50,000 feet
ESnet
CERN etc.
NLR
ORNL Connector
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Network Overview
OC192
10GbE
10GbE
OC192
10GbE
OC192
10GbE
OC192
CI
2 x OC192
CHI – ATL 2 x OC192
Tx ETF
CHI – ORNL
Ultranet
OLA - 2
OLA2
OLA - 3
OLA1
SOADM conversion
10GbE
OC192
OLA -1
Oakridge
Memphis
CI
GbE / 10GbE / OC102
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
20 x Gig-E???
Knoxville
10GbE
OC192
Atalnta
Acworth
Dalton
Chattanooga
Stevenson
Christiana
Nashville
Bowling Green
Upton
Louisville
Seymour
Indianapolis
Crawfordsville
Monon
Schneider
Chicago
Intro 10GLAN PHY – CD
10GbE
OC192
The Logical View
P NNL
Dedicat ed Lambdas, NLR
Chicago
MSP P
SONET swit ch
MSP P
Dedicat ed Lambdas, NLR
ORNL
MSP P
Scheduled Lambdas, ESnet
SONET ADM
Sunnyvale
MSP P
At lant a
MSP P
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
SONET ADM
SONET swit ch
Schedule
 Oak Ridge Connector to be up in June
 Fall-back circuits via NLR if necessary
 Chicago-Sunnyvale paced by NLR
 PNNL fiber schedule will pace their connection
 Expect first user-traffic in June
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Future Plans
 ORNL is half the team building CHEETAH




Nagi Rao and Malathi Veeraraghavan are PI’s
Funded by NSF
Will Connect ORNL, Atlanta, NC State, possibly CUNY
Explicitly developing signaling protocols
 Want to partner with HOPI
 …and all of you
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Thank You
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY