Introduction to the ACTS Collection Why/How Do We Use These Tools? Tony Drummond Computational Research Division Lawrence Berkeley National Laboratory University of California, Berkeley -

Download Report

Transcript Introduction to the ACTS Collection Why/How Do We Use These Tools? Tony Drummond Computational Research Division Lawrence Berkeley National Laboratory University of California, Berkeley - 

Introduction to the ACTS Collection
Why/How Do We Use These Tools?
Tony Drummond
Computational Research Division
Lawrence Berkeley National Laboratory
University of California, Berkeley - CS267
April 20, 2006 - Berkeley, California
Introduction to The
ACTS Collection
Lesson =
High Quality Software Reusability
Library Development
Funded by DOE/ASCR
Numerical Tools
Code Development
Run Time Support
http://acts.nersc.gov
General Purpose Libraries
•Algorithms
•Data Structures
•Code Optimization
• Programming Languages
•O/S - Compilers
Hardware - Middleware - Firmware
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
2
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Systems of
Linear Equations
Algorithms
Library
LU
Factorization
ScaLAPACK(dense)
SuperLU (sparse)
Cholesky
Factorization
ScaLAPACK
LDLT (Tridiagonal ScaLAPACK
Direct Methods
matrices)
QR
Factorization
ScaLAPACK
QR with column
ScaLAPACK
pivoting
LQ factorization ScaLAPACK
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
3
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Systems of
Linear Equations
(cont..)
Iterative
Methods
Algorithms
Library
Conjugate
Gradient
AztecOO (Trilinos)
PETSc
GMRES
AztecOO
PETSc
Hypre
CG Squared
AztecOO
PETSc
Bi-CG Stab
AztecOO
PETSc
Quasi-Minimal
Residual (QMR)
AztecOO
Transpose Free
QMR
AztecOO
PETSc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
4
Structure of PETSc
PETSc PDE Application Codes
ODE Integrators
Visualization
Nonlinear Solvers,
Interface
Unconstrained Minimization
Linear Solvers
Preconditioners + Krylov Methods
Object-Oriented
Grid
Matrices, Vectors, Indices
Management
Profiling Interface
Computation and Communication Kernels
MPI, MPI-IO, BLAS, LAPACK
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
5
Hypre Conceptual Interfaces
Linear System Interfaces
Linear Solvers
GMG, ...
FAC, ...
Hybrid, ...
AMGe, ...
ILU, ...
unstruc
CSR
Data Layout
structured
composite
block-struc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
6
Hypre Conceptual Interfaces
INTERFACE TO SOLVERS
List of Solvers and Preconditioners per Conceptual Interface
System Interfaces
Solvers
Jacobi
SMG
PFMG
BoomerAMG
ParaSails
PILUT
Euclid
PCG
GMRES
Struct
X
X
X
X
X
X
X
X
X
SStruct
FEI
IJ
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
X
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
7
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Systems of
Linear Equations
(cont..)
Iterative
Methods
(cont..)
Algorithms
Library
SYMMLQ
PETSc
Precondition CG
AztecOO
PETSc
Hypre
Richardson
PETSc
Block Jacobi
Preconditioner
AztecOO
PETSc
Hypre
Point Jocobi
Preconditioner
AztecOO
Least Squares
Polynomials
PETSc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
8
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Systems of Linear
Equations
(cont..)
Algorithms
SOR Preconditioning
Overlapping Additive
Schwartz
Iterative
Methods
(cont..)
Approximate Inverse
Sparse LU
preconditioner
PETSc
PETSc
Hypre
AztecOO
PETSc
Hypre
Incomplete LU (ILU)
preconditioner
AztecOO
Least Squares
Polynomials
PETSc
MG Preconditioner
MultiGrid (MG)
Methods
Library
Algebraic MG
PETSc
Hypre
Hypre
Hypre
Semi-coarsening
The DOE ACTS
Collection (http://acts.nersc.gov)
04/20/06
9
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Linear Least
Least Squares
Squares Problems
Standard
Eigenvalue Problem
Singular Value
Problem
Generalized
Symmetric Definite
Eigenproblem
Minimum Norm
Solution

Minimum Norm
Least Squares

Symmetric
Eigenvalue Problem


Singular Value

Decomposition
Eigenproblem

Algorithm
min x || b Ax || 2
min x || x || 2
Library
ScaLAPACK
ScaLAPACK
min x || b Ax || 2
min x || x || 2
ScaLAPACK
Az  z
ScaLAPACK (dense)
SLEPc (sparse)
For A=AH or A=AT
A  UVT
A  UV H
Az  Bz
ABz  z
BAz  z
The
DOE ACTS Collection (http://acts.nersc.gov)
ScaLAPACK (dense)
SLEPc (sparse)
ScaLAPACK (dense)
SLEPc (sparse)
04/20/06
10
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Non-Linear
Equations
Newton Based
Algorithm
Library
Line Search
PETSc
Trust Regions
PETSc
Pseudo-Transient
Continuation
PETSc
Matrix Free
PETSc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
11
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Non-Linear
Optimization
Newton Based
CG
Direct Search
Algorithm
Library
Newton
OPT++
TAO
Finite-Difference
Newton
OPT++
TAO
Quasi-Newton
OPT++
TAO
Non-linear Interior
Point
OPT++
TAO
Standard Nonlinear CG
OPT++
TAO
Limited Memory
BFGS
OPT++
Gradient
Projections
TAO
No derivate
information
OPT++
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
12
TAO - Interface with PETSc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
13
OPT++ Interfaces
• Four major classes of problems available
• NLF0(ndim, fcn, init_fcn, constraint)
• Basic nonlinear function, no derivative information
available
• NLF1(ndim, fcn, init_fcn, constraint)
• Nonlinear function, first derivative information available
• FDNLF1(ndim, fcn, init_fcn, constraint)
• Nonlinear function, first derivative information
approximated
• NLF2(ndim, fcn, init_fcn, constraint)
• Nonlinear function, first and second derivative
information available
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
14
ACTS Numerical Tools: Functionality
Computational
Problem
Methodology
Non-Linear
Optimization
Newton Based
CG
Direct Search
Algorithm
Library
Newton
OPT++
TAO
Finite-Difference
Newton
OPT++
TAO
Quasi-Newton
OPT++
TAO
Non-linear Interior
Point
OPT++
TAO
Standard Nonlinear CG
OPT++
TAO
Limited Memory
BFGS
OPT++
Gradient
Projections
TAO
No derivate
information
OPT++
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
15
ACTS Numerical Tools: Functionality
Computational
Problem
Non-Linear
Optimization
(cont..)
Ordinary
Differential
Equations
Nonlinear
Algebraic
Equations
Differential
Algebraic
Equations
Methodology
Algorithm
Library
Feasible
Semismooth
TAO
Unfeasible
semismooth
TAO
Integration
Adam-Moulton
(Variable coefficient
forms)
CVODE (SUNDIALS)
CVODES
Backward
Differential Formula
Direct and Iterative
Solvers
CVODE
CVODES
Line Search
KINSOL (SUNDIALS)
Direct and Iterative
Solvers
IDA (SUNDIALS)
Semismoothing
Inexact Newton
Backward
Differential Formula
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
16
ACTS Numerical Tools: Functionality
Computational
Problem
Support
Writing Parallel
Programs
Techniques
Shared-Memory
Global Arrays
Distributed
Memory
CUMULVS (viz)
Globus (Grid)
Grid Generation
OVERTURE
Distributed Arrays Structured Meshes
Semi-Structured
Meshes
GRID
Distributed
Computing
Library
CHOMBO (AMR)
Hypre
OVERTURE
PETSc
CHOMBO (AMR)
Hypre
OVERTURE
Globus
Remote Steering
CUMULVS
Coupling
PAWS
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
17
ACTS Numerical Tools: Functionality
Computational
Problem
Writing Parallel
Programs (cont.)
Support
Distributed
Computing
Profiling
Algorithmic
Performance
Execution
Performance
Code
Optimization
Library
Installation
Interoperability
Technique
Library
Check-point/restart CUMULVS
Automatic
instrumentation
PETSc
User
Instrumentation
PETSc
Automatic
Instrumentation
TAU
User
Instrumentation
TAU
Linear Algebra
Tuning
ATLAS
Language
BABEL
CHASM
Components
CCA
Code Generation
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
18
Introduction to The
ACTS Collection
User Interfaces
CALL BLACS_GET( -1, 0, ICTXT )
CALL BLACS_GRIDINIT( ICTXT, 'Row-major', NPROW, NPCOL )
:
CALL BLACS_GRIDINFO( ICTXT, NPROW, NPCOL, MYROW, MYCOL )
:
:
CALL PDGESV( N, NRHS, A, IA, JA, DESCA, IPIV, B, IB, JB, DESCB,
$
INFO )
Library Calls
•
•
Command lines
More advanced:
•
•
•
Linear System Interfaces
FAC
Hybrid, ...
-ksp_max_it <max_iters>
-ksp_gmres_restart <restart>
-pc_asm_overlap <overlap>
Problem Domain
Linear Solvers
GMG
-ksp_type
[cg,gmres,bcgs,tfqmr,…]
-pc_type
[lu,ilu,jacobi,sor,asm,…]
AMGe
ILU, ...
unstruc
CSR
Data Layout
structured
composite
blockstrc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
19
Introduction to The
ACTS Collection
User Interfaces
PyACTS
Ax = b
matlab*P Star-P
User
Az  z
View_field(T1)
NetSolve
A UV
T
High Level Interfaces
OPT++
AZTEC
ScaLAPACK
PAWS
Hypre
SuperLU
Globus
PETSc
TAO
CUMULVS
Chombo
PVODE
TAU
Global Arrays
Overture
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
20
Tool Interoperability
Tool-to-Tool
PETSc
TAU
Ex 1
Ex 2
TOOL A
Trilinos
TOOL D
Ex 3
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
21
Component Technology!
Tool A
Tool C
Tool B
Tool D
CCA
ESI
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
22
Introduction to PETSc
• Introducción a PETSC
• Vectors
• Matrices
• PETSc viewers
• Use of Preconditioners in PETSc
• Some examples using PETSc
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
23
What is PETSc? What is PETSc?
PETSc:
Portable Extensible Toolkit for Scientific Computation
• Developed at Argonne National Laboratory
• Library that supports work with PDE
• Public Domain
• Portable to a variety of platforms
• Project started in 1991.
• Written in C with a fortran interface and OO flavor
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
24
PETSc Structure
PETSc PDE Application Codes
ODE Integrators
Visualization
Nonlinear Solvers,
Interface
Unconstrained Minimization
Linear Solvers
Preconditioners + Krylov Methods
Object-Oriented
Grid
Matrices, Vectors, Indices
Management
Profiling Interface
Computation and Communication Kernels
MPI, MPI-IO, BLAS, LAPACK
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
25
Some Algorithmic Implementations in PETSc
Nonlinear Solvers
Newton-based Methods
Line Search
Time Steppers
Other
Backward Pseudo Time
Euler
Stepping
Euler
Trust Region
Other
Krylov Subspace Methods
GMRES
Additive
Schwartz
CG
Block
Jacobi
CGS
Bi-CG-STAB
Richardson Chebychev
TFQMR
Other
Preconditioners
Jacobi
ILU
ICC
LU
(Sequential only)
Others
Matrix-free
Other
Matrices
Compressed
Sparse Row
(AIJ)
Blocked Compressed
Sparse Row
(BAIJ)
Block
Diagonal
(BDIAG)
Distributed Arrays
Dense
Index Sets
Indices
Block Indices
Stride
Other
Vectors
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
26
Vectors and Matrices in PETSc
Vectors
Fundamental Objects to
store fields, right-hand side
vectors, etc. .
Matrices
Fundamental Objects to store
operators (like Jacobians)
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
27
Basic Vector Operations
•
•
Each process owns a piece of the vector
VECTOR Types: Sequential, MPI or SHARED
proc 0
–VecCreate(MPI_Comm Comm,Vec * v)
• comm - MPI_Comm parallel processes
• v = vector
–VecSetType(Vec,VecType)
• Valid Types are:
–VEC_SEQ, VEC_MPI, o
VEC_SHARED
–VecSetSizes(Vec *v,int n, int N)
 Where n or N (not both) can be
PETSC_DECIDE
– VecDestroy(Vec *)
The DOE ACTS Collection (http://acts.nersc.gov)
proc 1
proc 2
proc 3
proc 4
04/20/06
28
PETSc - Vector Example
(in C) Operations
Basic Vector
#include petscvec.h
int main(int argc,char **argv)
{
Vec
x;
int
n = 20,m=4, ierr;
PetscInitialize(&argc,&argv);
VecCreateMPI(PETSC_COMM_WORLD, m, n, x);
VecCreate(PETSC_COMM_WORLD,&x);
VecSetSizes(x,PETSC_DECIDE,n);
VecSetFromOptions(x);
<-- USER_CODE:Work with PETSc VECTOR -->
PetscFinalize();
return 0;
}
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
29
Basic Operations with Vectors
Function Name
Operation
VecAXPY(Scalar *a, Vec x, Vec y)
y = y + a*x
VecAYPX(Scalar *a, Vec x, Vec y)
y = x + a*y
VecWAXPY(Scalar *a, Vec x, Vec y, Vec w) w = a*x + y
VecScale(Scalar *a, Vec x)
x = a*x
VecCopy(Vec x, Vec y)
y=x
VecPointwiseMult(Vec x, Vec y, Vec w)
w_i = x_i *y_i
VecMax(Vec x, int *idx, double *r)
r = max x_i
VecShift(Scalar *s, Vec x)
x_i = s+x_i
VecAbs(Vec x)
x_i = |x_i |
VecNorm(Vec x, NormType type , double *r) r = ||x||
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
30
Basic Matrix Operations
Types:
•
•
•
•
•
•
default sparse AIJ: MPIAIJ, SEQAIJ
block sparse AIJ (for multi-component PDEs): MPIAIJ, SEQAIJ
symmetric block sparse AIJ: MPISBAIJ, SAEQSBAIJ
block diagonal: MPIBDIAG, SEQBDIAG
dense: MPIDENSE, SEQDENSE
matrix-free
–MatCreate(MPI_Comm Comm, Mat *A)
• MPI_Comm - group of process that work on A
–MatSetSizes(Mat A, PetscInt m, PetscInt n, PetscInt M,
PetscInt N)
–MatSetType(Mat,MatType)
–MatDestroy(Mat)
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
31
Basic Matrix Operations
•
Single user interface to matrices (operators)
• Input values to a PETSc matrix
• MatSetValues(Mat mat,PetscInt m,const PetscInt
idxm[],PetscInt n,const PetscInt idxn[],const
PetscScalar v[],InsertMode addv)
• Matriz-vector multiplication
For instance: MatMult(A,y,x) (y≠x)
• Matrix viewing
•
• MatView()
Multiple underlying implementations
• AIJ, block AIJ, symmetric block AIJ, block diagonal, dense, matrix-free, etc.
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
32
Basic Matrix Operations
Block-Row distribution per process
proc 1
M=8,N=8,m=3,n=k1
rstart=0,rend=4
proc 2
M=8,N=8,m=3,n=k2
rstart=3,rend=6
proc 3
M=8,N=8,m=2,n= k3
rstart=6,rend=8
The values of k1, k2 y k3 depend on the application and must be compatible with vector x (Ax=b)
MatGetOwnershipRange(Mat A, int *rstart, int *rend)
• rstart: first global row number that belongs to a process
• rend -1: Last row global number that belongs to a process
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
33
PETSc Matrix Example
Included in PETSc Distribution:
1) $PETSC_DIR/src/mat/tutorial/ex2.c
2) $PETSC_DIR/src/mat/tutorial/ex5.c
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
34
Basic Viewer Operations
•
Obtain basic information on PETSc viewers
•
Run time options
•
Extract data that can be later use by other packages/app. programs.
• vector fields, matrix contents
• various formats (ASCII, binary)
•
Visualization
• simple graphics created with X11 to display vector campo, non-zero element structure of
a matrix, etc.
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
35
Basic Viewer Operations
• MatView(Mat A, PetscViewer v);
• Runtime options available after matrix assembly
-mat_view_info Information related to matrix type
-mat_view_draw nonzero pattern
-mat_view
etc.
data in ASCII
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
36
Examples of Viewers and Matrices
Included in PETSc Distribution:
1) $PETSC_DIR/src/mat/tests/ex2.c
1) Using the -mat_view_info_detailed, etc
2) $PETSC_DIR/src/mat/tests/ex3.c
1) Using the -mat-view-draw
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
37
PETSc KSP
Objective: Support the solution of Linear System of the form,
Ax=b,
Krylov Subspace methods are highly dependent of the spectrum of the matrix A.
The use of preconditioners can speed-up their convergence.
KRYLOV SUBSPACE METHODS + PRECONDITIONERS
R. Freund, G. H. Golub, and N. Nachtigal. Iterative Solution of Linear Systems,pp 57-100.
ACTA Numerica. Cambridge University Press, 1992.
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
38
Right vs. Left Preconditioning
Ax=b,
1
(ML1AM1
)(M
x)

M
R
R
L b,
For ML=I
For MR=I
r  b  AMR1 MR x
1
1
rL  M 1
L b  ML Ax  M L r
PETSC
Default
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
39
Using PETSc’s KSP
Main Routine
PETSc
Linear Solvers (SLES)
Solve
Ax = b
Application
Initialization
PC
KSP
Evaluation of A and b
User code
PostProcessing
PETSc code
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
40
Solvers and Preconditioners in PETSc
Krylov Methods (KSP)
Preconditioners (PC)
•
Conjugate Gradient
•
Block Jacobi
•
GMRES
•
Overlapping Additive Schwarz
•
CG-Squared
•
ICC, ILU via BlockSolve95
•
Bi-CG-stab
•
ILU(k), LU
•
Transpose-free QMR
•
etc.
•
Chebychev
•
CGNE
•
Many more (check manual)
Interface to Other Packages
(like SuperLU, SuperLU_DIST,
Hypre, etc)
KSPPREONLY
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
41
Flow of Control for PDE Solution
Main Routine
Timestepping Solvers (TS)
Nonlinear Solvers (SNES)
Linear Solvers (KSP)
PETSc
PC
Application
Initialization
Function
Evaluation
User
code
Jacobian
Evaluation
PostProcessing
PETSc code
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
42
ODE Solvers (TS) in PETSc
Time Stepper Types (TS)
TS_EULER - Euler
TS_SUNDIALS- SUNDIALS interface
TS_BEULER - Backward Euler
TS_PSEUDO- Pseudo-timestepping
•Linear TS Example (ex3.c)
•Non Linear TS Example (ex1.c)
The DOE ACTS Collection (http://acts.nersc.gov)
04/20/06
43