Transcript Document
Simulations of Large Earthquakes on the Southern San Andreas Fault Amit Chourasia Visualization Scientist San Diego Supercomputer Center Presented to: Latin American Journalists July 11, 2007 Global Seismic Hazard Source: Global Seismic Hazard Assessment Program Growth of Earthquake Risk Growth of cities 2000-2015 Expansion of urban centers in tectonically Increasing Loss active areas is driving an exponential increase in earthquake risk. Source: National Geographic Slide: Courtesy Kim Olsen Risk Equation Risk = Probable Loss (lives & dollars) = Hazard Faulting, shaking, landsliding, liquifaction Exposure Fragility Extent & density of built environment Structural vulnerability Slide: Courtesy Kim Olsen Seismic Hazard Analysis Definition: Specification of the maximum intensity of shaking expected a site during a fixed time interval Example: National seismic hazard maps (http://geohazards.cr.usgs.gov/eq/) • Intensity measure: peak ground acceleration (PGA) • Interval: 50 years • Probability of exceedance: 2% Slide: Courtesy Kim Olsen The FEMA 366 Report “HAZUS’99 Estimates of Annual Earthquake Losses for the United States”, September, 2000 • U.S. annualized earthquake loss (AEL) is about $4.4 billion/yr. • For 25 states, AEL > $10 million/yr • 74% of the total is concentrated in California • 25% is in Los Angeles County alone Slide: Courtesy Kim Olsen Southern California: a Natural Laboratory for Understanding Seismic Hazard and Managing Risk Tectonic diversity Complex fault network High seismic activity Excellent geologic exposure Rich data sources Large urban population with densely built environment high risk Extensive research program coordinated by Southern California Earthquake Center (SCEC) under NSF and USGS sponsorship Slide: Courtesy Kim Olsen 1994 Northridge When: 17 Jan 1994 Where: San Fernando Valley Damage: $20 billion Deaths: 57 Injured: >9000 Slide: Courtesy Kim Olsen Major Earthquakes on the San Andreas Fault, 1690-present 146+91-60 yrs 1906 M 7.8 Slip deficit on the southern SAF since last event (1690): 315 years x 16 mm/year = 5.04 m -> Mw7.7 1857 M 7.9 ~1690 M 7.7 220±13 yrs Slide: Courtesy Kim Olsen TeraShake Simulation Region 600km x 300km x 80km Spatial resolution = 200m Mesh Dimensions 3000 x 1500 x 400 = 1.8 billion mesh points Simulated time = 4 minutes Number of time steps = 22,728 (0.011 sec time step) 60 sec source duration from Denali 3D Crustal structure: subset of SCEC CVM3.0 Near-surface S-wave velocity truncated at 500m/s, up to 0.5 Hz Computational Challenge! TeraShake-2 Data Flow Okaya Initial 200m 200m Media Stress modify TS2.dyn.200m 30x 256 procs, 12 hrs, Initial 100m Okaya SDSC IA-64 TG IA-64 Stress modify 100m Media GPFS TS2.dyn.100m TG IA-64 10x 1024 procs, 35 hrs GPFS-wan NCSA IA-64 100m Reformatting GPFS NCSA-SAN Network 100m Transform SDSC-SAN 100m Filtering 200m moment rate TS2.wav.200m 3x 1024 procs, 35 hrs Datastar Datastar p690 Velocity mag. & cum peak GPFS Datastar p655 Displace. mag & cum peak Seismograms SRB HPSS Visualization SAM-QFS Analysis Registered to Digital Library Slide: Courtesy Yifeng Cui Challenges for Porting and Optimization Before Optimization After Optimization Code deals up to 24 million mesh nodes Codes enhanced to deal with 32 billion mesh nodes Code scales up to 512 processors Excellent speed-up to 40,960 processors, 6.1 Tflop/s Ran on local clusters only Ported to p655, BG/L, IA-64, XT3, Dell Linux etc No checkpoints/restart capability Added Checkpoints/restart/checksum capability Wave propagation simulation only Integrated dynamic rupture + wave propagation as one Researcher’s own code Serve as SCEC Community Velocity Model Mesh partition and solver in one Mesh partition separated from solver Initialization not scalable, large memory need 10x speed-up of initialization, scalable, memory reduced I/O not scalable, not portable MPI-I/O improved 10x, scaled up to 40k processors TeraShake code Total Execution Time on IBM Power4 Datastar Wall Clock Time (sec, 101 steps) 10000.00 So urce: 600x300x80km M esh: 3000x1500x400 Spatial reso lutio n: 200m Number o f steps: 101 Output: every time step 1000.00 95% 86% efficiency 100.00 WCT time with impro ved I/O WCT ideal WCT time with TeraShake-2 WCT time with TeraShake-1 86% 10.00 120 240 480 Number of processors 960 1920 Slide: Courtesy Yifeng Cui Data from TeraShake 1.1 Scalar Surface (floats) Scalar Volume (floats) • 3000 x 1500 ie 600 km x 300 km =17.2 MB per timestep • 20,000 timesteps • 3 variables Vx, Vy & Vz Velocity components • Total Scalar data = 1.1 TB • 3000 x 1500 x 400 ie 600 x 300 x 80 km^3 =7.2 GB per timestep • 2,000 timesteps • 3 variables Vx, Vy & Vz Velocity components • Total Vol data = 43.2 TB Other Data – check points,etc Grand Total = 47.4 TB Aggregate Data : 160 TB (seven simulations) Visualization Movie (1.5 mb) Comparative Visualization Movie (11 mb) Scenario Comparison PGV (NW-SE Rupture) PGV (SE-NW1 Rupture) Topography Deformation Movie (11 mb) Glimpse of Visualization Movie (65 mb) Visualization Over 130,0000 images Consumed 40,000 hrs of compute time More than 50 unique animations Does Viz work? Does Viz work? TeraShake Results TeraShake-1 • • NW-directed rupture on southern San Andreas Fault is highly efficient in exciting L.A. Basin Maximum amplification from focusing associated with waveguide contraction • Peak ground velocities exceeding 100 cm/s over much of the LA basin • Uncertainties related to simplistic source description. TeraShake-2 • • • • Extremely nonlinear dynamic rupture propagation Effect of 3D velocity structure: SENW and NW-SE dynamic models NOT interchangeable Stress/strength/tapering - weak layer required in upper ~2km to avoid super-shear rupture velocity Dynamic ground motions: kinematic pattern persists in dynamic results, but peak motions 50-70% smaller than the kinematic values due to less coherent rupture front Slide: Courtesy Yifeng Cui Summary TeraShake demonstrated that optimization and enhancement of major applications codes are essential for using large resources (number of CPUs, number of CPU-hours, TBs of data produced) TeraShake showed that multiple types of resources are needed for large problems: initialization, run-time execution, analysis resources, and long-term collection management TeraShake code as a community code now used by the wider SCEC community Significant TeraGrid allocations are required to advance the seismic hazard analysis to a more accurate level Next: PetaShake! Slide: Courtesy Yifeng Cui References Chourasia, A., Cutchin, S. M., Olsen, K.B., Minster, B., Day, S., Cui, Y., Maechling, P., Moore, R., Jordan, T. (2007) “Visual insights into high-resolution earthquake simulations”, IEEE Computer Graphics & Applications (Discovering the Unexpected) Sept-Oct 2007, In press. Cui, Y., Moore, R., Olsen, K., Chourasia, A., Maechling, P., Minster. B., Day, S., Hu, Y., Zhu, J., Majumdar, A., Jordan, T. (2007), Enabling very-large scale earthquake simulations on parallel machines "Advancing Science and Society through Computation", International Conference on Computational Science 2007, Part I, Lecture Notes in Computer Science series 4487, pp. 46-53, Springer Olsen, K.B., S.M. Day, J.B. Minster, Y. Cui, A. Chourasia, M. Faerman, R. Moore, P. Maechling, and T. Jordan (2006). Strong shaking in Los Angeles expected from southern San Andreas earthquake, Geophys. Res. Lett. 33, L07305,doi:10.1029/2005GRL025472 TeraShake Collaboration Large Scale Earthquake Simulation on Southern San Andreas 33 researchers, 8 Institutions Southern California Earthquake Center San Diego Supercomputer Center Information Sciences Institute Institute of Geophysics and Planetary Physics (UC) University of Southern California San Diego State University University of California, Santa Barbara Carnegie-Mellon University ExxonMobil Slide: Courtesy Marcio Faerman Acknowledgements Southern California Earthquake Center (SCEC) San Diego Supercomputer Center (SDSC) Funding: National Science Foundation Thanks for your patience Q&A Websites: http://www.sdsc.edu/us/sac (Computation) http://epicenter.usc.edu/cmeportal/TeraShake.html (Seismology) http://visservices.sdsc.edu/projects/scec/terashake (Visualization)