Transcript Solution Accurate, Efficient and Stable Unsplit MHD Solvers in FLASH

The Flash Center for Computational Science
A Solution Accurate, Efficient and Stable
Unsplit Staggered Mesh MHD Solver in
FLASH
Dongwook Lee
University of
Chicago
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Outline
 Split vs. unsplit formulations
 Unsplit solvers in FLASH (UHD & USM)
 CFL stability (reduced or full?)
 Reduced/Full corner-transport-upwind (CTU) for 3D
 Divergence-free magnetic fields for USM-MHD
 constrained-transport (CT)
 Verifications, convergence, performance
 Runtime parameters
 Summary
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 1
Dimensionally Split vs. Unsplit???
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 1
 Single-mode Rayleigh-Taylor Instability
 Top figures:
Dimensionally split using PLM, PPM+old
limiter, PPM+new limiter
high-wavenumber instabilities grow
Bottom figures:
 Dimensionally unsplit using PLM,
PPM+old limiter, PPM+new limiter
high-wavenumber instabilities
suppressed
 the split solvers experience high
compressions and expansions in subsequent
directional sweeps where there is a local high
strain rate
 Almgren et al, ApJ, 715, 2010
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 1
 Weakly magnetized 2D field loop
 Gardiner and Stone 2005 (JCP); Lee and Deane 2009 (JCP)
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 1
 8-wave split MHD
scheme (Powell et al.
1999) at t=2.0
 Unsplit staggered
mesh MHD scheme
(Lee and Deane, 2009)
at t=2.0
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 1
 What is wrong with the split formulation for MHD?
 In the split formulation, you cannot correctly include terms proportional to
 Gardiner and Stone (2005)
 Dynamics of in-plane magnetic fields in x and y directions are ruined from
erroneous growth of magnetic field in z direction:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
Unsplit Hydro/MHD Solvers & Algorithms
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Hydro Unit in FLASH
Hydro_Unsplit
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Unsplit Staggered Mesh (USM) MHD Solver
 Shock-capturing high-order Godunov Riemann solver (Lee & Deane, JCP,
2009; Lee 2012, to be submitted)
 Finite volume method
 New data reconstruction-evolution algorithm for high-order accuracy
 Adaptive mesh refinement, uniform grid
 1st order Godunov, 2nd order MUSCL-Hancock, 3rd order PPM, 5th Order
WENO
 Approximate Riemann solvers: Roe, HLL, HLLC, HLLD, Marquina, modified
Marquina, Local Lax-Friedrichs
 Monotonicity preserving upwind PPM slope limiter for MHD (Lee, 2010,
Astronum)
 Divergence of magnetic fields is numerically controlled on a staggered grid,
using a constrained transport (CT) method (Evans & Hawley, 1998)
 Wide ranges of plasma flows
 Full Courant stability limit (CFL ~ 1 for 3D)
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Unsplit Formulations
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
MHD Governing Equations
 MHD system of equations:
 This can be written in a simple matrix form:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
MHD Governing Equations
 Conservative variables and fluxes:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Linearized System
 A primitive form:
where the coefficient matrix is
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Corner Transport Upwind (CTU)
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Corner Transport Upwind (CTU)
Normal predictor
Transverse corrector
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Corner Transport Upwind (CTU)
 Traditional approach (Colella 1990;
Saltzman 1994)
Characteristic tracing for the
normal predictor
Subsequent calls to Riemann
solvers for transverse corrector
Normal predictor
Transverse corrector
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Corner Transport Upwind (CTU)
 Traditional approach (Colella 1990;
Saltzman 1994)
Characteristic tracing for the
normal predictor
Subsequent calls to Riemann
solvers for transverse corrector
 New approach (Lee and Deane
2009):
Characteristic tracing for BOTH
normal predictor and transverse
corrector!
Normal predictor
Transverse corrector
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Linearized System, cont’d
 A primitive form:
where the coefficient matrix is
 First consider the evolution in the x-normal direction and treat the normal
magnetic field separately from the other variables:
 Normal predictor
 MHD source term
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Single-step data Reconstruction-evolution in USM
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Characteristic tracing for Transverse corrector
 A jump relationship:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Reduced 3D CTU in USM
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Full 3D CTU in USM
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Summary of Part 1
 New approach of using characteristic tracing for BOTH normal predictor and
transverse corrector
 Reduced 3D CTU
 A direct extension of 2D CTU to 3D
 Requires 3 Riemann solves for 3D (6-ctu needs 6 Riemann solves)
 Only including second cross derivatives
 CFL limit ~ 0.5
 Full 3D CTU
 Full considerations of accounting for third cross derivatives
 Requires 3 Riemann solves for 3D (12-ctu needs 12 Riemann solves)
 CFL limit ~ 1.0
 20% relative performance gain compared to reduced 3D CTU
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
Divergence-Free fields:
Constrained Transport (CT) MHD
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
 CT scheme by Balsara and Spicer, 1998:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2: recall…
 Conservative variables and fluxes:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
 New upwind biased modified electric field construction(upwind-MEC), Lee
2012:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
 Small angle advection of the 2D field loop:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 2
 Small angle advection of the 3D field loop:
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Summary of Part 2
 Three CT schemes were discussed:
 Standard CT scheme by Balsara and Spicer, 1998:
 Takes a simple arithmetic averaging
 Lacks numerical diffusion for magnetic fields advection
 Modified electric field construction (MEC) scheme by Lee and Deane,
2009:
 3rd order accurate in space
 Not enough numerical diffusion for field advection
 Upwind biased MEC (upwind-MEC) scheme by Lee, 2012 (to be
submitted)
 Upwind scheme of MEC
 Added numerical diffusion to stabilize field advection
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
Verification, convergence, and performance
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Part 3
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Summary of Part 3
 Verification tests for the reduced/full 3D CTU schemes:
 CFL=0.95 for all 3D simulations using the full CTU scheme
 CFL=0.475 for the reduced CTU scheme
 They both converge in 2nd order
 20% performance gain in using the full CTU scheme:
CPU F-ctu
» 0.8
CPU R-ctu
 Various choices in runtime parameters
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Conclusion
 Directionally split vs. unsplit formulations for hydro and MHD
 Unsplit hydro/MHD solvers in FLASH4 (also FLASH3 in part)
 The reduced and full 3D CTU algorithms
 Upwind-MEC scheme for MHD
 Stable solutions with 2nd order convergence with CFL=0.95
 20% performance gain in the full CTU scheme over the reduced CTU
scheme
 Work in progress:
 Fully implicit Jacobian-Free Newton-Krylov implicit solver for the unsplit
solvers
 More HEDP capabilities for the USM solver
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Thank You
Questions?
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
New Upwind PPM for Slowly Moving Shock
larger By
Standard PPM
Upwind PPM
Standard PPM with increasing By
5th order WENO
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
New Upwind PPM for Slowly Moving Shock
larger By
Standard PPM
Standard PPM with increasing By
Lee, 2010, 5th Astronum Proceeding;
Lee, 2011, in preparation
Upwind PPM
5th order WENO
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Block and Mesh Packages
q
q
q
Mesh package can be selected at configuration
time
The basic abstraction is a block of interior cells
surrounded by guard cells
Grid unit makes sure that blocks are self
contained before being given to the solvers
Oct tree based AMR AMR with variable patch
PARAMESH
size - CHOMBO
FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Uniform Grid