Transcript Performance of Applications using Dual-Rail
Performance of Applications Using Dual-Rail InfiniBand 3D Torus Network on the Gordon Supercomputer
Dongju Choi, Glenn Lockwood, Robert Sinkovits, Mahidhar Tatineni San Diego Supercomputer Center University of California, San Diego SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Background
• SDSC data intensive supercomputer Gordon: • 1,024 dual-socket Intel Sandy Bridge nodes, each with 64 GB DDR3–1333 memory • 16 cores per node and 16 nodes (256 cores) per switch • Large IO nodes and local/global ssd disks • Dual rails QDR InfiniBand network supports IO and Compute communication separately.
• Can be scheduled to be used for computation also.
• We have been interested witch communication oversubscription in switch-to-switch and switch/node topology effects on application performance.
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Gordon System Architecture
3-D torus of switches on Gordon Subrack level network architecture on Gordon
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
MVAPICH2 MPI Implementation
• •
MVAPICH2 current version 1.9, 2.0 on the Gordon system Full control of dual rail usage at the task level via user settable environment variables:
• MV2_NUM_HCAS=2, • MV2_IBA_HCA=mlx4_0:mlx4_1 • MV2_RAIL_SHARING_LARGE_MSG_THRESHOLD=8000: can be as low as 8KB, • MV2_SM_SCHEDULING=ROUND_ROBIN: explicitly distribute tasks over rails
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
OSU Micro-Benchmarks
• • •
Compare the performance of single and dual rail QDR InfiniBand vs FDR InfiniBand: evaluate the impact of rail sharing, scheduling, and threshold parameters Bandwidth tests Latency tests SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
OSU Bandwidth Test Results for Single Rail QDR, FDR, and Dual-Rail QDR Network Configurations - Single rail FDR performance is much better than single rail QDR for message sizes larger than 4K bytes - Dual rail QDR performance exceeds FDR performance at sizes greater than 32K - FDR showing better performance between 4K and 32K byte sizes due to the rail-sharing threshold
OSU Bandwidth Test Performance with
MV2_RAIL_SHARING_LARGE_MSG_THRESHOLD=8K - Lowering the rail sharing threshold bridges the dual-rail QDR, FDR performance gap down to 8K bytes .
OSU Bandwidth Test Performance with
MV2_SM_SCHEDULING = ROUND_ROBIN - Adding explicit round-robin tasks to communicate over different rails
OSU Latency Benchmark Results for QDR, Dual Rail QDR with MVAPICH2 Defaults, FDR
- There is no latency penalty at small message sizes (expected as only one rail is active below the striping threshold). - Above the striping threshold a minor increase in latency is observed but the performance is still better than single rail FDR.
OSU Latency Benchmark Results for QDR, Dual-Rail QDR with Round Robin Option, FDR
- Distributing messages across HCAs using the round-robin option increases the latency at small message sizes.
- Again, the latency results are better than the FDR case.
Application Performance Benchmarks
•
Applications
• P3DFFT Benchmark • LAMMPS Water Box Benchmark • AMBER Cellulose Benchmark •
Test Configuration
• Single Rail vs. Dual Rails • Multiple Switch Runs with Maximum Hops=1 or no hops limit for 512 core runs (2 switches are involved)
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
P3DFFT Benchmark
• • •
Parallel Three-Dimensional Fast Fourier Transforms Used for studies of turbulence, climatology, astrophysics and material science Depends strongly on the available bandwidth as the main communication component is driven by transposes of large arrays (alltoallv) SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Simulation Results for P3DFFT benchmark with 256 cores and QDR, Dual-Rail QDR
Run# QDR Wallclock Time (s) Dual-Rail QDR Wallclock Time (s)
- Dual-rail runs are consistently faster than the single rail runs, with an average performance gain of 23%. 1 2 3 4 992 985 991 993 761 760 766 759
Communication and Compute Time Breakdown for 256 core, Single/Dual QDR rail P3DFFT Runs.
Run # 1 2 3 4
Single Rail Runs
Total Time
992
Comm. Time
539
Compute Time
453 985 535 450 991 539 452 993 543 450
Run # 1 2 3 4
Dual Rail Runs
Total Time
761
Comm. Time
302
Compute Time
459 760 766 301 308 459 458 759 300 459 - Compute part is nearly identical in both sets of runs - Performance improvement is almost entirely in the communication part of the code - Shows that Dual rail boosts the alltoallv performance and consequently speeds up the overall calculation
Communication and Compute Time Breakdown for 512 core, Single/Dual QDR Rail P3DFFT Runs. Maximum Switch Hops=1
Run # 1
Single Rail Runs
Total Time Comm. Time Compute Time
802 592 210
Run # 1
Dual Rail Runs
Total Time
537
Comm. Time
322
Compute Time
215
2
802 592 210
2
538 322 216
3
804 594 210
3
538 322 216
4
803 592 211
4
538 322 216 - Shows similar dual rail benefits - Fewer runs pans/links, reducing the likelihood of oversubscription due to other jobs - Also can increase the likelihood of oversubscription due to lesser switch connections
P3DFFT benchmark with 512 cores, Single Rail QDR. No Switch Hop Restriction
Run # 1 2 3 4 5 6 Total Time
717 732 789 726 825 697
Comm. Time
506 525 580 518 615 488
Compute Time
211 207 209 208 210 209 - oversubscription is mitigated by topology of the run and the performance is nearly 15% better than the single hop case. However, as seen from the results a different topology may also lead to lower performance if the distribution is not optimal (it could be by oversubscription of the job itself or from other jobs).
P3DFFT benchmark with 512 cores, Single Rail QDR. No Switch Hop Restriction
Run # 1..3
4 5 6 Total Time
726 825 697
Comm. Time
518 615 488
Compute Time
208 210 209 - Spread out the computation on several switches. Lowering bandwidth requirements on a given set of switch-to-switch links - bad for latency bound codes (given the extra switch hops) but benefit bandwidth sensitive codes depending on the topology of the run - Nukada et. al. utilizes dynamic links to minimize congestion to perform better in the dual-rail case Nukada, A., Sato, K. and Matsuoka, S.. 2012. Scalable multi-GPU 3-D FFT for TSUBAME 2.0 supercomputer.
In
Proceedings of the International Conference on HighPerformance Computing, Networking, Storage and Analysis
(SC '12). IEEE Computer Society Press, LosAlamitos, CA, USA, Article 44, 10 pages.
Communication and Compute Time Breakdown for 1024 Core P3DFFT Runs
Run
Single Rail Runs
Total Time Comm . Time Compute Time 1 2
404 408 307 310 97 98
Run
Dual Rail Runs
Total Time Comm. Time Compute Time 1 2
332 325 232 226 100 99 -. No switch hop restrictions are placed on the runs. -. Communication aspect is greatly improved in the dual rail cases while compute fraction is the nearly identical in all the runs.
LAMMPS Water Box Benchmark
• • •
Large-scale Atomic/Molecular Massively Parallel Simulator (LAMMPS) is a widely used classical molecular dynamics code. 12,000 water molecules (36,000 atoms) are set in the input Simulation is run for 20 picoseconds. SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
LAMMPS Water Box Benchmark with Single/Dual Rail QDR and 256 cores.
Run #
1 2 3 4
QDR Wallclock Time (s)
57 57 58 57
Dual-Rail QDR Wallclock Time (s)
46 46 46 46 - Dual-Rail runs show better performance than the single rail runs and mitigate communication overhead with an average of 32% in wallclock time used.
improvement
LAMMPS Water Box Benchmark with Single Dual Rail QDR and 512 Cores
Run # Single Rail QDR w MAX_HOP=1 Wallclock Time (s) Single Rail QDR w No Limit in MAX_HOP Wallclock Time (s) Dual Rail QDR Wallclock Time (s)
1 2 3 4 69 69 70 69 71 70 281 450 47 47 48 47 - Application is not scaling due to larger communication overhead (happens due to fine level of domain decomposition) - LAMMPS benchmark is very sensitive to topology and shows large variations if the maximum switch hops are not restricted
AMBER Cellulose Benchmark
• •
Amber is a package of programs for molecular dynamics simulations of proteins and nucleic acids. 408,609 atoms are used for the tests. SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Amber Cellulose Benchmark with Single/Dual Rail QDR and 256 Cores
Run #
1 2 3 4
Single Rail QDR Wallclock Time (s)
218 219 218 219
Dual Rail QDR Wallclock Time (s)
212 213 212 212 - Communication overhead is low and the dual rail benefit is minor (<3%)
Amber Cellulose Benchmark with Single/Dual Rail QDR, 512 cores
Run # Single Rail QDR w MAX_HOP=1 Wallclock Time (s) Single Rail QDR w No Limit in MAX_HOP Wallclock Time (s) Dual Rail QDR Wallclock Time (s) 1 204 332 168 2 3 4 202 202 202 331 396 373 168 168 167
- There is a modest benefit (<5 %) in the single rail QDR runs - Communication overhead increases with increased core count, leading to the drop off in scaling. This can be mitigated with dual rail QDR - Dual rail QDR performance is better by 17%
Amber Cellulose Benchmark with Single/Dual Rail QDR, 512 cores
Run #
1 2 3 4
Single Rail QDR w MAX_HOP=1 Wallclock Time (s)
204 202 202 202
Single Rail QDR w No Limit in MAX_HOP Wallclock Time (s)
332 331 396 373
Dual Rail QDR Wallclock Time (s)
168 168 168 167 - Dual rail enables the benchmark to scale to higher core count - Shows sensitivity to the topology due to the larger number of switch hops and possible contention from other jobs
Summary
• • • •
Aggregated bandwidth obtained with dual rail QDR exceeds the FDR performance. Shows performance benefits from dual rail QDR configurations.
Gordon’s 3-D torus of switches leads to variability in performance due to oversubscription/topology considerations. Switch topology can be configured to enable mitigation of the link oversubscription bottleneck.
SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Summary
• •
Performance improvement also varies based on the degree of communication overhead.
• Benchmark cases with larger communication fractions (with respect to overall run time) show more improvement with dual rail QDR configurations.
Computational time scaled with the core counts in both single and dual rail configurations for the currently benchmarked applications: LAMMPS and Amber SAN DIEGO SUPERCOMPUTER CENTER at the UNIVERSITY OF CALIFORNIA, SAN DIEGO
Acknowlegements
• This work was supported by NSF grant: OCI #0910847 Gordon: A Data Intensive Supercomputer.