EU Network Meeting June 2001 Cactus

Gabrielle Allen, Tom Goodale Max Planck Institute for Gravitational Physics, (Albert Einstein Institute)

Outline

     Brief what is Cactus?

Review of developments since last meeting    Current release Beta 10 New Viz Tools, thorns, architectures Current work General Relativity thorns  Released thorns in CactusEinstein    Other relevant thorns Matter thorns from the WashU Relativity Group Matter in Cactus: current status and plans Rest of the meeting  Talk with me and Tom about what you want to use Cactus for  Talk with me if you want to work through some examples, make sure your laptop has everything installed right, get some advice on how to start, etc Any time left … demos of new viz/control stuff

Cactus in a Nutshell

     Cactus acts a the “main” routine of your code, it takes care of e.g. parallelism, IO, checkpointing, parameter file parsing for you (if you want), and provides different computational infrastructure such as reduction operators, interpolators, coordinates, elliptic solvers, … Everything Cactus “does” is contained in thorns (modules), which you need to compile-in. If you need to use interpolation, you need to add in a thorn which does interpolation.

It is very extensible, you can add you own interpolators, IO methods etc.

Not all the computational infrastructure you need will be there, for example we don’t have a thorn for 2D coordinates yet, you may need to add your own (then hopefully make it available for others to use).

We’re trying to provide a easy-to-use environment for collaborative , high-performance computing, from easy compilation on any machine, to easy visualization of your output data.

Porting Your Code

program YourCode call ReadParameterFile call SetUpCoordSystem call SetUpInitialData call OutputInitialData do it=1,niterations call EvolveMyData call AnalyseData call OutputData end do end

If your existing code has this kind of structure   Split into subroutines Clear argument lists it should be straightforward to put into Cactus.

Cactus will take care of the parameter file, and (hopefully) the coord system and IO, and if you’re lucky you can take someone elses Analysis, Evolution, Initial Data, … modules.

Thorn EvolveMyData

  Probably you will make a thorn to do the evolution. This thorn will simply contain the source code of your existing evolution routines (with hopefully not to many changes).

You will need to add configuration files which detail how your thorn interacts with the Flesh and other thorns.

Thorn EvolveMyData Parameter Files and Test Suites Configuration Files ????

Source Code ????

Fortran Routines Documentation!

C Routines C++ Routines Make Information

Black Hole Run

Physics Computational Extract Exact PsiKadelia AHFinder IDAnalyticBH PUGHInterp IOBasi c PUG H IOASCII PUGHReduce ADMConstraints CartGrid3D IOUtil ADM Einstein MyThorn FLESH

Parameter File

ActiveThorns = "Time CartGrid3D PUGH pughslab pughreduce Einstein IDAnalyticBH ADM IOASCII IOUtil iobasic" cactus::cctk_itlast = 160 grid::type = "byspacing" grid::dx = 0.15

grid::dy = 0.15

grid::dz = 0.15

grid::domain = "octant" driver::global_nx = 20; driver::global_ny = 20; driver::global_nz = 20; einstein::initial_lapse = "one" einstein::evolution_system = "ADM" einstein::initial_data = "schwarzschild" einstein::slicing = "1+log" adm::bound = "radiative" adm::method = "stagleap" IO::outdir = "einstein1" IOASCII::out1D_vars = "einstein::grr einstein::alp" IOASCII::out1D_every = 5 IOBasic::outInfo_vars = "einstein::grr" IOBasic::outInfo_every = 20

How To Find Out About Thorns

     Look at web pages   http://www.cactuscode.org/Documentation/Thorns/ AEI and WashU thorns ??

Ask a colleague for a sample parameter file  New checkout script to make it easy to check out Parameters?

  Look in param.ccl in any thorn Use command line options from executable e.g.

./exe/cactus_ -O Thorn documentation:  gmake -ThornGuide  gmake ThornGuide No documentation? Complain to authors.

Current Release: Beta 10

      New thorns:   CactusUtils/NaNChecker CactusPUGH/PUGHInterp (replaces CactusPUGH/Interp) Flesh:  Changes to time levels, interpolation API, parameters (more error checking for parameter file) Thorns:  Boundary conditions, initialize memory to NaN/Zero/uninitialized Tools:  Recombiner for HDF5 data New architectures:  MacOS X (no Fortran or MPI) Thorn documentation  gmake -ThornGuide

Planned Development

       Easier scripts for checking out and compiling   GetCactus works now from a ThornList Adding GetCactus from a parameter file, and also automatically getting configuration file for a know machine Einstein:  Improve spacetime – matter interface so that any thorns can be used together for spacetime, matter evolution   Add timelevels (gxx, gxx_p, gxx_p_p), should improve performance, and be ready for AMR Easier viz, set up networks for OpenDX, Amira Release Cactus 4.0 !!

 Mainly documentation and odd and ends Elliptic Solvers:  Get what is there (SOR, PETSc, ) fully working and better documented using the elliptic interfaces Coordinates: standard 2D, 1D, spherical Cool stuff: web interface, EMail/SMS thorn, lots more … Ask for what you need, what is difficult: [email protected]

From Other Projects

     AMR  Good progress with FMR and AMR now Portal  Web interface from anywhere to all your machines, parameter files, collaborative and other awesome stuff Grid Computing  For us this really means not needing to know anything at all about the machines you have access to, codes automatically seek out and exploit your resources Visualization  Remote visualization very important for big simulations  Also trying to make it easier, scripts to automatically find the right viz clients and networks for data (VizLauncher), networks specifically for GR Data description and management  Again important for big simulations being run on many different machines  Archiving data also really important, understanding and being able to reproduce old data sets

Users View

Computational Toolkit

CactusBase  Boundary, IOUtil, IOBasic, CartGrid3D, IOASCII, Time CactusBench  BenchADM CactusConnect  HTTPD, HTTPDExtra CactusExample  WaveToy1DF77, WaveToy2DF77 CactusElliptic  EllBase, EllPETSc, EllSOR, EllTest CactusPUGH  PUGHInterp, PUGH, PUGHSlab, PUGHReduce CactusPUGHIO  IOFlexIO, IOHDF5, IsoSurfacer CactusIO  IOJpeg CactusTest  TestArrays, TestCoordinates, TestInclude1, TestInclude2, TestComplex, TestInterp, TestReduce CactusWave  IDScalarWave, IDScalarWaveC, IDScalarWaveCXX, WaveBinarySource, WaveToyC, WaveToyCXX, WaveToyF77, WaveToyF90, WaveToyFreeF90 CactusExternal  FlexIO, jpeg6b CactusUtils  NaNChecker

Einstein Toolkit

 Thorn CactusEinstein/Einstein    Sets up standard GR variables for 3D problems: – gxx, gxy, gxz, gyy, gyz, gzz, – kxx,kxy,kxz,kyy, kyz, kzz, – alp, betax, betay, betaz Other tools – Mixed slicing methods (us maximal every 10 timesteps or if ????) – Courant timestep condition – Calculate spherical components – Generic mask function You need to use these variables if you want to use any of the other Einstein thorns or thorns from AEI/WashU

Einstein Toolkit

   Spacetime evolution:  ADM: staggered leapfrog, iterative Crank-Nicholson Analysis:  AHFinder: locates apparent horizons using either minimization or flow algorithm.

   ADMConstraints: violation of constraint equations.

PsiKadelia: calculate gauge invariant quantities Extract: waveforms from nearly axisymmetric, nearly linear regions Initial Data:  IDAnalyticBH, IDBrillData,IDLinearWaves,IDAxiBrillBH

Other Thorns

     Links for more information on these will be added to the EU web pages AEI:     Web Page?

ADM_BSSN: BS evolution with different shift conditions Lazarus: NP quantities, perturbative evolution Excision: masks out region inside a black hole AMR:   Carpet PAGH Zeus:  Newtonian MHD Alpha/BetaThorns  Maximal, Cartoon2D

Now Tom …

 Tom will talk about what’s available at WashU, how Matter is currently treated with the CactusEinstein thorns, and how it is going to be improved so that we can use different spacetime and matter evolvers together.

More Information ...

     Look at news on Cactus home page www.cactuscode.org

Also lots more information on Cactus pages:  Worked example of WaveToy showing how to use web server thorn, viz clients Mailing lists: news, users, developers CVS updates of the thorns you are using Email cactusmaint.cactuscode.org (or me, Tom, Ed, ..)

Dynamic Grid Computing

Add more resources SDSC Free CPUs!! LRZ Queue time over, find new machine RZG Go!

Archive data Clone job with steered parameter Calculate/Output Invariants Found a horizon, try out excision Look for horizon Calculate/Output Grav. Waves Find best resources NCSA SDSC