Transcript UltraNet Engineering - University of Virginia

Enabling Supernova Computations by
Integrated Transport and Provisioning Methods
Optimized for Dedicated Channels
Nagi Rao, Bill Wing, Tony Mezzacappa
Oak Ridge National Laboratory
Malathi Veeraraghavan
University of Virginia
DOE MICS PI Meeting: High-Performance Networking Program
September 14-16, 2004
Fermi National Laboratory
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
1
Outline
 Background
 ORNL Tasks
 Preliminary Results
 UVA Tasks
 Preliminary Results
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Terascale Supernova Initiative - TSI
 Science Objective: Understand supernova evolutions
 DOE SciDAC Project: ORNL and 8 universities
 Teams of field experts across the country collaborate on
computations
 Experts in hydrodynamics, fusion energy, high energy
physics
 Massive computational code
 Terabyte/day generated currently
 Archived at nearby HPSS
 Visualized locally on clusters – only archival data
 Current Networking Challenges
 Limited transfer throughput
 Hydro code – 8 hours to generate and 14 hours to transfer
out
 Runaway computations
 Find out after the fact that parameters needed adjustment
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
TSI Desired Capabilities
Data and File Transfers (terabyte – petabyte)
 Move data from computations on supercomputers
 Supply data to visualizations on clusters and supercomputers
Interactive Computations and Visualization
 Monitor, collaborate and steer computations
 Collaborative and comparative visualizations
Visualization channel
Visualization control channel
Steering channel
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Computation or
visualization
Background on NSF CHEETAH Project
 Circuit-switched High-speed End-to-End Transport
arcHitecture (CHEETAH)
 Team: UVA, ORNL, NCSU, CUNY
 Concept:
 Share bandwidth on a dynamic call-by-call basis
 End-to-end circuit:
 Ethernet - Ethernet over SONET - Ethernet
 Network
 Second NICs at hosts in a compute cluster/viz cluster
 Connected to MSPPs that perform Ethernet-SONET mapping
 GMPLS-enabled SONET crossconnects
 Transport protocols and middleware
 To support file transfers on dedicated circuits
 To support remote visualization and computational steering
 Applications to support TSI scientists
 SFTP
 Ensight + new visualization programs
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Current DOE ORNL-UVA Project:
Complementary Roles
•Project Components:
•Provisioning for UltraScience Net - GMPLS
•File transfers for dedicated channels
•Peering – DOE UltraScience Net and NSF CHEETAH
•Network optimized visualizations for TSI
•TSI application support over UltraScience Net + CHEETAH
ORNL
Visualization
TSI Application
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
UVA
Peering
Provisioning
File Transfers
This project leverages two projects
•DOE UltraScience Net
•NSF CHEETAH
Peered UltraScienceNet-CHEETAH
Enables coast-to-coast dedicated channels
Phase I: TL1-GMPLS cross conversion
Phase II: GMPLS-based
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
ORNL: Year 1 Activities
•
•
•
Peering CHEETAH - UltraScienceNet
Visualization
• Decomposable visualization pipeline
• Analytical formulation
• First implementation
TSI support
• Monitoring Visualizations
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
ORNL Personnel
Nagi Rao, Bill Wing, Tony Mezzacappa (PIs)
Qishi Wu (Post-Doctoral Fellow)
Mengxia Zhu (Phd Student – Louisiana State Uni.)
Publications
Conference Papers
•
M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “Adaptive Visualization Pipeline Partition and
Mapping on Computer Network”, International Conference on Image Processing and Graphics,
ICIG2004.
•
M. Zhu, Q. Wu, N. S. V. Rao, S. S.Iyengar, “On Optimal Mapping of Visualization Pipeline
onto Linear Arrangement of Network Nodes”, International Conference on Visualization and
Data Analysis, 2005
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Modules of Visualization Pipeline
Data
source
raw data
filtering
filtered data
transformation
(topological surface
construction, volumetric
transfer function)
transformed data
(geometric model,
volumetric values)
rendering
framebuffer
Display

Visualization Modules
 Pipeline consists of several modules
 Some modules are better suited to certain network nodes
 Visualization clusters
 Computation clusters
 Power walls
 Data transfers between modules are of varied sizes and rates
Note:
Commercial tools do not support efficient decomposition
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Grouping Visualization Modules
G1
G2
Gq-1
mu-1
mv-1
Mu-1
Mu
Mv-1
Mw
c1
cu-1
cu
cv-1
cw
bs
,P[2]
ps
vP[2]
bP[2],P[3]
pP[2]

mx-1
M1
vs
Gq
Mx-1
cx-1
Mx
cx
b P[q-1],d
vP[q-1]
Mn+1
cn+1
vd
pd
pP[q-1]
Grouping
 Decompose the pipeline into modules
 Combine the modules into groups
 Transfers on single node are generally faster
 Between node transfers take place over the network
 Align bottleneck network links between modules with least data
requirements
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Optimal Mapping of Visualization Pipeline:
Minimization of Total Delay
q
q 1
Ttotal ( Path P of q nodes)  Tcomputing  Ttransport  TGi  TLP[ i ],P[ i1]
i 1
i 1
 1
 

i 1  pP[ i ]
q
 
jGi and j  2
c j m j 1

 q 1  m(Gi )
   
 i 1  bP[i ], P[i 1]
Dynamic Programming Solution
 Combine modules into groups
 Align bottleneck network links between modules with least
data requirements
 Polynomial-time solvable O(n  E ) – not NP-complete
 T m 1 (v)  cm mm

pv

m
T (v)  min 
m 1 to n , vV
 min ( T m 1 (u )  cm mm  cm mm
)
p
b
u

adj
(
v
)

v
u ,v








Note:
1. Commercial tools (Ensight) are not readily amenable to optimal
network deployment
2. This method can be implemented into tools that provide appropriate
hooks
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY



Optimal Mapping of Visualization Pipeline:
Maximization of Frame Rate
Tbottleneck ( Path P of q nodes )

max
Path P of q nodes
i 1,2, , q 1
T
computing

(Gi ), Ttransport ( LP[i ], P[i 1] ), Tcomputing (Gq )
 1

 c j m j 1 ,
 pP[i ] jGi and j  2
 m(G )

i
 max 
,
Path P of q nodes b
P
[
i
],
P
[
i 1]
i 1,2, , q 1 
 1

 c j m j 1
 pP[ q ] jGq and j  2














(2)
Dynamics Programming Solution
 Align bottleneck network links between modules with least
data requirements
 Polynomial-time solvable O(n  E ) – not NP-complete

 m 1
GS m 1  vi   cm mm  

max  F (vi ),
,
p


v
i

m


F (vi )  min 
m 1 ton , vi V


 m 1 cm mm

mm
min
max
F
(
u
),
,
uadj (vi ) 

pvi
bu ,vi  



OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY



 (6)



First Implementation
NCSU
ORNL
Slavenode
Computer
Slavenode
Headnode
Slavenode
Slavenode
LSU
Computer

Client/Server OpenGL implementation (leveraged from CHEETAH)
 Case 1: small cube geometry or frame-buffer
 Case 2: small geometry
 Case 3: small geometry
 CT scan: raw image or frame-buffer
Dimension
Estimated
bandwidth
Minimum
delay
Raw data
size/delay
Geometry
size/delay
FB size/delay
Cube
1
10x6x8
0.284Mbps
0.032sec
8 K / 0.257sec
1K / 0.032sec
1.8M/50.73sec
Cube
2
50x20x39
0.300Mbps
0.034sec
610K / 16.3sec
16K / 0.46sec
1.8M/48.03sec
Cube
3
150x210x1
39
0.277Mbps
0.033sec
71.6M / 34.4min
2.4M / 69.34sec
1.8M/52.01sec
Hand
256x256x8
0
0.239Mbps
0.033sec
81.9M /
45.69min
NA
1.8M/60.28sec
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
Monitoring Visualization
 Requirements
 Light-weight server located at the computation site
 Remote client provides constant monitoring of variables
 Our first implementation
 OpenGL server and client
 Client
 Geometric operations
 Point, iso-surface, vector view
 Commercial Visualization tools
 Not light weight – server on supercomputers
 Expensive – collaborative visualization by team
 Not optimized for network deployment
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY
ORNL: Year 2 Activities
•
•
•
MPLS Peering CHEETAH
Visualizations
• Computational Monitoring
• Collaborative Visualization
TSI support
• Collaborative Steering
• Integrated Data Transfers
OAK RIDGE NATIONAL LABORATORY
U. S. DEPARTMENT OF ENERGY