Transcript Framework Design and Implementation of Finite Difference

Framework Design and Implementation of
Finite Difference-Based Seismic Simulations
Igor Terentyev
Department of Computational and Applied Mathematics
Rice University, Houston, TX
[email protected]
February 29, 2008
Igor Terentyev
1994 - 2000
MS, Moscow State University
2001 - 2005
Schlumberger
2006 - present
Rice University, CAAM
Scientific advisor: Prof. William Symes
Internships:
ExxonMobil, 2007, 2008
Acknowledgements:
Dr. William Symes
Dr. Tanya Vdovina
Igor Terentyev
TRIP Annual Meeting 02/29/2008
2
Outline
Forward problem
Model problem
Numerical methods
Computational cost
Numerical solver
Problem scope
Software requirements
Software
Parallelization
Scheme coupling
Framework
Igor Terentyev
TRIP Annual Meeting 02/29/2008
3
Model Problem
The acoustic wave equation:
1 p
 t    v  f p

 v  p  f
v
 t
p(x,t), v(x,t) are pressure and velocity,
f (x,t) is the source of acoustic energy,
 (x),  (x) are medium parameters.
Numerical method: finite differences (e.g. staggered, 2-2k)

Computational cost:
15x15x8 km domain of interest ~ 109 grid points
20 FLOP / grid point
1000 time steps
50000 simulations
10
18
FLOP ~ 30 years to simulate a typical seismic
survey on a 1GFLOPS desktop!
Igor Terentyev
TRIP Annual Meeting 02/29/2008
4
Problem Scope
Simple geometry (rectangular shape) domains
Large scale
1D, 2D, 3D
Time-dependent finite-difference methods
Explicit schemes
Uniformly spaced grids
Regular
Staggered
Boundary conditions
Free surface
Absorbing (PML)
Igor Terentyev
TRIP Annual Meeting 02/29/2008
5
Motivation
Misc. code exists…
Open source?
Verification/benchmarks?
The SEG Advanced Modeling Project (SEAM)
http://www.seg.org/seam/
Goals:
Design and generate synthetic model 3D and 2D seismic data
Share the high cost effort of substantial model design and generation
Provide a forum to discuss geophysical problems of interest
Advance the art of modeling and computation
Provide data sets for industry benchmarks and educational purposes
Igor Terentyev
TRIP Annual Meeting 02/29/2008
6
Solver Requirements
Various (user-provided) FD methods
Coupled FD schemes
FRAMEWORK
specific design
Parallel
hybrid: MPI + multicore (OpenMP, …)
Portable
ISO C90
Max speedup
User-friendly
careful implementation
helper data structures / routines
Open source
https://cvs.caam.rice.edu:3129/svn/tsoptroot/trunk/seamx
Igor Terentyev
TRIP Annual Meeting 02/29/2008
7
Parallelization
Approach: data domain decomposition
Goal: hide sync. from user
Processor 1
Global data
m
m+1
Synchronize
m
m
m+1
m+1
Processor 2
Igor Terentyev
TRIP Annual Meeting 02/29/2008
8
Scheme coupling
10
unkn.
p,
vx,
vy,
vz,
px,
vx
p, vx, vy, vz, py,vy
Physical
NPML
p, vx, vy, vz
Issues:
• Interaction between schemes
• Data exchange
Igor Terentyev
TRIP Annual Meeting 02/29/2008
9
Parallelization Strategy
Approaches
[ uhn1  Lh uhn ]
Per processor models
Single global model
Framework provides:
Framework provides:
Base types: Matrix, Vector
Base methods: Multiply(A,b)
Data exchange functionality
Types: ParMatrix, ParVector
Methods: ParMultiply(A,b)
User provides:
Data: Lh, uh
Hidden synchronization, but
hard to combine different
models


Efficiency
Igor Terentyev
User provides:
Model(s): MyMatrix, MyVector
Can combine different
models, but need model
communication info

TRIP Annual Meeting 02/29/2008
10
Virtual Array
Stored
array
Storage ptr
Storage ptr
Local frame
Local frame
Virt.arr.1
Virt.arr.2
Methods:
Create/resize subarrays
Output
Etc.
Igor Terentyev
TRIP Annual Meeting 02/29/2008
11
Virtual Domain
Multiple models: different variables, FD schemes, grids.
Uniform data structure, time-step function.
RDOM - collection of arrays and rules for a model.
Examples: 2D Staggered grid.
Physical
Arrays : p,  , v x ,  x , v y ,  y
NPML
p,  , v x ,  x , v y ,  y , p x , p y , v x , v y
p(i, j)  p(x, y)

 (i, j)   (x, y)
Rules : 

v x (i, j)  v x (x   x /2, y)

v y (i, j)  v y (x, y   y /2)
Time step function works on RDOM

Igor Terentyev
TRIP Annual Meeting 02/29/2008
12
Model
RARR
RDOM
TSFUN(RDOM, na)
STENCIL
MODEL
User provides:
Computational RDOM
Stencil
Time-step function
Time-step sequence
Model provides:
Allocated and exchanged areas
Setting MPI exchange data types
Automating data exchange
Igor Terentyev
TRIP Annual Meeting 02/29/2008
13
Exchange automation
STENCIL
p
vx
vy
Proc. L
RDOMs:
Igor Terentyev
Computational
Receive
Allocated
Send
TRIP Annual Meeting 02/29/2008
Proc. R
2/8/26 neighbors
14
Time-step Loop
Performed by the framework
Based on recomputed array sequence (MODEL)
[ p, v x , v y , vz ]
Recomputed

SEND
CENTER
For [time-step loop]
For each array in sequence
Compute SEND frame
Start SEND/RECV
Compute CENTRAL area
Finish SEND/RECV
End
End
Igor Terentyev
TRIP Annual Meeting 02/29/2008
15
Results
Timings
2D
Staggered grid
2-2 and 2-4 schemes
2375 x 2400 grid
1999 time steps
Itanium, 2-2 Itanium, 2-4 Opteron, 2-2 Opteron, 2-4
# of
T
proc (sec)
1 1243
2 629
4 315
8 159
16
85
# of
T
proc (sec)
1 1502
2
757
4
380
8
191
16
101
32
51
# of
T
proc (sec)
1 214
2 131
4
78
8
40
16
20
# of
T
proc (sec)
1 249
2 126
4
82
8
49
16
23
32
14
14.6
29.5
10.7
17.8
Igor Terentyev
TRIP Annual Meeting 02/29/2008
16
Summary
Software framework for FD-based PDE solvers developed
First tests performed
To do:
Finish and cleanup
More tests
Documentation
Igor Terentyev
TRIP Annual Meeting 02/29/2008
17