Transcript WRF Architecture

WRF Model: Software
Architecture
WRF Tutorial, 8/16/01
John Michalakes, NCAR
Overview
• Purpose
– Familiarization with concept and structure behind WRF
software from the point of view of a scientific
user/developer
• Outline
– Code organization
– Framework goals and design aspects
– Software hierarchy, model interface, and memory
model
– Parallelism
– Data structures
– The WRF Registry
Directory Structure (WRF 1.1)
clean*
compile*
configure*
Registry/
Registry
arch/
configure.defaults
external/
IOAPI
RSL/
io_netcdf/
inc/
src/
test/
b_wave/
hill2d_x/
quarter_ss/
real/
squall2d_x/
squall2d_y/
tools/
use_registry
clean script
compile script
configure script
registry file
arch-specific compile options
WRF I/O API specification
external comm package: RSL
external I/O package: NetCDF
holds intermediate include files
source directory (.F files)
directory containing test cases
Perl scripts implementing Registry
Directory Structure (WRF 1.2)
clean*
compile*
configure*
Registry/
Registry
arch/
configure.defaults
dyn_eh/
dyn_em/
dyn_slt/
external/
frame/
inc/
main/
wrf.F
phys/
share/
test/
tools/
registry*
clean script
compile script
configure script
registry file
arch-specific compile options
Eulerian-height dyncore source files
Eulerian-mass
dyncore source files
semi-implicit semi-Lagrangian dyncore
same as 1.1
driver layer (framework) source files
holds intermediate include files
main WRF source file
physics directory
files shared across dyncores
test cases
new implementation of registry mechanism
WRF Model Software
Goals
• Good performance
• Portable across a range of
architectures
• Flexible, maintainable,
understandable
• Facilitate code reuse
• Multiple
dynamics/physics options
• Run-time configurability
• Package independence
Aspects of Design
• Single-source code
• Fortran90 modules,
dynamic memory,
structures, recursion
• Hierarchical design
• Multi-level parallelism
• CASE: Registry
• Package APIs
model
mediation
driver
WRF main:
• Top-level flow of control
- Initialize packages
- Input config info
- Allocate, initialize,
decompose main domain
• Start integration
• Shutdown
Code structure
Solve_interface:
• 1 step on 1 domain
• Select dyncore
• Dereference ADT
Top-level model layer
Top-level model layer
subroutines:
Top-level model layer
subroutines:
• One
tile computation
• subroutines:
One tile computation
• Boundary conditions/avoidance
•Boundary
One tileconditions/avoidance
computation
• •May
select physics opt.
•May
Boundary
select conditions/avoidance
physics
opt.
• •May
dereference
4D fields
•
May
select
physics
opt.
• May dereference 4D fields
• May dereference 4D fields
Integrate:
• Time loop
• Nesting (recursive)
• Calls to I/O
Solve_eh:
• Solve_eh:
Sequence through
tile loops
•calling
Sequence
through
loops
model layertile
routines
calling
model
routines
for
dynamics
andlayer
physics
for dynamics and
physics
• Multi-threading
(OpenMP)
Multi-threading
(OpenMP)
• •Interprocessor
communi•cation
Interprocessor communication
model
Code structure
• Top-level flow of control
- Initialize packages
- Input config info
Integrate:
WRF domain derived data type
(frame/module_domain.F)
- Allocate, initialize,
• Time loop
decompose main domain
• Nesting (recursive)
MODULE module_domain
• Start integration
• Calls to I/O
• Shutdown
TYPE (domain)
generated
by Registry
mediation
driver
WRF main:
REAL, DIMENSION(:,:,:), POINTER :: eh_ru_1
Solve_eh:
REAL, DIMENSION(:,:,:), POINTER
:: eh_ru_2
Solve_interface:
Solve_eh:
•
Sequence
through tile loops
REAL, DIMENSION(:,:,:), POINTER :: eh_rv_1
• 1 step on 1 domain
•calling
Sequence
through
loops
model
layertile
routines
. . .
• Select dyncore
calling
model
routines
for
dynamics
andlayer
physics
END TYPE (domain)
• Dereference ADT
for dynamics and
physics
• Multi-threading
(OpenMP)
Multi-threading
(OpenMP)
• •Interprocessor
communiCONTAINS
• Interprocessor communiSUBROUTINE allocate_space_field (cation
grid , . . . )
cation
TYPE (domain), POINTER :: grid
IF ( dyn_opt == DYN_EH ) THEN
Top-level
model
layer
ALLOCATE(
grid%eh_ru_1(ims:ime,kms:kme,jms:jme)
Top-level
model
layer
subroutines:
ALLOCATE(
grid%eh_ru_1(ims:ime,kms:kme,jms:jme)
Top-level
model
layer
subroutines:
• One
tile
. . computation
.
subroutines:
•
One
tile
computation
• Boundary
conditions/avoidance
ELSE
•
One
tile
computation
Boundary
conditions/avoidance
• •May
select
physics opt.
.
.
.
•May
Boundary
select conditions/avoidance
physics
opt.
• •May
dereference
4D fields
•
May
select
physics
opt.
• May dereference 4D fields
• May dereference 4D fields
Code structure
driver
WRF main:
• Top-level flow of control
- Initialize packages
pseudo
for info
integration (frame/module_integrate.F)
- Inputcode
config
Integrate:
- Allocate, initialize,
• Timeend_step
loop
RECURSIVE
SUBROUTINE
( grid, start_step,
)
decompose
mainintegrate
domain
• Nesting (recursive)
• Start integration
• Calls to I/O
DO
step = start_step , end_step
• Shutdown
model
mediation
CALL solver ( grid )
WHILE ( active nests
Solve_interface:
! Advance 1 step
of grid )
Solve_eh:
• Solve_eh:
Sequence through
tile loops
• 1 step on 1 domain
•calling
Sequence
through
loops
model layertile
routines
CALL force_nest( grid, grid->child(i),
• Select dyncore
calling
model
routines
for
dynamics
andlayer
physics
mapping, subset, interpolator
)
• Dereference ADT
for dynamics and
physics
• Multi-threading
(OpenMP)
CALL integrate ( grid->child(i),
Multi-threading
(OpenMP)
• •Interprocessor
communi(step-1)*(nest_ratio**nest_level),
•cation
Interprocessor communi(step)*(nest_ratio**nest_level)
cation
CALL fdbck_nest( grid->child(i), grid,
mapping, subset, interpolator )
END
Top-level model layer
Top-level model layer
subroutines:
WHILE
Top-level model layer
subroutines:
• One
tile computation
• subroutines:
One tile computation
conditions/avoidance
END DO • Boundary
•Boundary
One tileconditions/avoidance
computation
•
• May select physics opt.
•May
Boundary
select conditions/avoidance
physics
opt.
• •May
dereference
4D fields
END SUBROUTINE
•
May
select
physics
opt.
• May dereference 4D fields
• May dereference 4D fields
Code structure
driver
WRF main:
• Top-level flow of control
- Initialize packages
pseudo code for solve interface
- Input config info
Integrate:
- Allocate, initialize,
(share/solve_interface.F)
• Time loop
decompose main domain
• Nesting (recursive)
• Start integration
SUBROUTINE interface ( grid
) to I/O
• Calls
• Shutdown
mediation
model
Solve_interface:
• 1 step on 1 domain
• Select dyncore
• Dereference ADT
( dyn_opt == DYN_EH ) THEN
generated
by Registry
IF
Solve_eh:
CALL solve_eh ( grid->eh_ru,
• Solve_eh:
Sequence through tile loops
grid->eh_rv,
•calling
Sequence through tile loops
grid->eh_rtb,model layer routines
calling
model
routines
for
dynamics
andlayer
physics
grid->eh_rtb,
for dynamics and
physics
• Multi-threading
(OpenMP)
grid->eh_rw,
• Multi-threading
(OpenMP)
communi.• .Interprocessor
.
•cation
Interprocessor communi)
cation
ELSE IF ( dyn_opt == DYN_EM ) THEN
. . )
DYN_SL ) THEN
. . )
ELSE IF ( dyn_opt
• Boundary conditions/avoidance
•
One
tile
computation
Boundary
. . opt.
.
• •May
select conditions/avoidance
physics
•
Boundary
conditions/avoidance
Maydereference
select physics
opt.
ENDIF
• •May
4D fields
•
May
select
physics
opt.
• May dereference
4D fields
END SUBROUTINE
• May dereference 4D fields
Top-level modelCALL
layer
solve_em ( .
Top-level model layer
subroutines:
IF (layer
dyn_opt ==
Top-level ELSE
model
subroutines:
CALL solve_sl ( .
• One
tile computation
• subroutines:
One tile computation
mediation
model
• Top-level flow (dyn_eh/solve_eh.F)
of control
- Initialize packages
SUBROUTINE
solve_eh
Input config
info (
ru, initialize,
rv, rtb, rtb, rw, . . .
- Allocate,
)
decompose main domain
• Start integration
dummy argument declarations
• Shutdown
i1 data declarations
Code structure
Integrate:
generated
by Registry
driver
WRF main:
pseudo code for eh solver
• Time loop
• Nesting (recursive)
• Calls to I/O
INTEGER, DIMENSION(max_tiles) ::
Solve_eh:
i_start, i_end, j_start, j_end
Solve_interface:
• Solve_eh:
Sequence through tile loops
CALLon
get_tiles
,
• 1 step
1 domain ( numtiles, i_start, i_end
•calling
Sequence
through
loops
model
layertile
routines
j_start, j_end calling
)
• Select dyncore
model
routines
for dynamics
andlayer
physics
.
.
.
• Dereference ADT
for dynamics and
physics
• Multi-threading
(OpenMP)
#include “HALO_EH_A.inc”
Multi-threading
(OpenMP)
• •Interprocessor
communi•cation
Interprocessor communi!$OMP DO PARALLEL
cation
DO ij = 1, numtiles
its = i_start(ij)
ite = i_end(ij)
Top-level
model ;layer
Top-level
model; jte
layer
jts = j_start(ij)
= j_end(ij)
subroutines:
Top-level
model
layer
CALL model_subroutine( arg1, arg2, . . .
• subroutines:
One
tile
subroutines:
ids
,computation
ide , jds , jde , kds , kde ,
•
One
tile
computation
• Boundary
conditions/avoidance
imstile
, conditions/avoidance
ime
, jms , jme , kms , kme ,
•Boundary
One
computation
• •May
select
physics
its
, ite
, jtsopt.
, jte , kts , kte )
•May
Boundary
conditions/avoidance
•
select
physics
opt.
•
May
dereference
4D
fields
END• DO
•May
Maydereference
select physics
opt.
4D fields
. . •. May dereference 4D fields
END SUBROUTINE
template for model layer subroutine
SUBROUTINE model ( &
arg1, arg2, arg3, …
ids, ide, jds, jde,
ims, ime, jms, jme,
its, ite, jts, jte,
, argn,
kds, kde,
kms, kme,
kts, kte
&
&
&
)
! Domain dims
! Memory dims
! Tile dims
•
•
IMPLICIT NONE
! Define Arguments (S and I1) data
REAL, DIMENSION (kms:kme,ims:ime,jms:jme) :: arg1, . . .
REAL, DIMENSION (ims:ime,jms:jme)
:: arg7, . . .
. . .
! Define Local Data (I2)
REAL, DIMENSION (kts:kte,its:ite,jts:jte) :: loc1, . . .
. . .
! Executable code; loops run over tile
! dimensions
DO j = jts, jte
DO i = its, ite
DO k = kts, kte
IF ( i > ids .AND. I < ide ) THEN
loc(k,i,j) = arg1(k,i,j) + …
ENDIF
END DO
END DO
END DO
•
Domain dimensions
• Size of logical domain
• Used for bdy tests, etc.
Memory dimensions
• Used to dimension dummy
arguments
• Do not use for local arrays
Tile dimensions
• Local loop ranges
• Local array dimensions
WRF Multi-Layer Domain Decomposition
Logical
domain
•
1 Patch, divided
into multiple tiles
Single version of code for efficient
execution on:
– Distributed-memory
– Shared-memory
– Clusters of SMPs
– Vector and microprocessors
Model domains are decomposed for parallelism on two-levels
Patch: section of model domain allocated to a distributed memory node
Tile: section of a patch allocated to a shared-memory processor within a
node; this is also the scope of a model layer subroutine.
Distributed memory parallelism is over patches; shared memory parallelism is
over tiles within patches
Inter-processor
communication
Distributed Memory Communications
•
•
•
•
dyn_eh/solve_eh.F
SUBROUTINE solve_eh ( & . . . )
IMPLICIT NONE
. . .
code before communication
#include “HALO_EH_A.incl”
code after communication
. . .
Generated by
Registry
Halo updates
Periodic boundary updates
Parallel transposes
Interface to code through
the mediation layer
Shared
Memory
Parallelism
pseudo code for eh solver
(dyn_eh/solve_eh.F)
SUBROUTINE solve_eh (
. . . )
USE module_tiles
. . .
INTEGER, DIMENSION(max_tiles) ::
i_start, i_end, j_start, j_end
CALL get_tiles ( numtiles, i_start, i_end ,
j_start, j_end )
. . .
!$OMP DO PARALLEL
DO ij = 1, numtiles
its = i_start(ij) ; ite = i_end(ij)
jts = j_start(ij) ; jte = j_end(ij)
CALL model_subroutine( arg1, arg2, . .
ids , ide , jds , jde , kds , kde
ims , ime , jms , jme , kms , kme
its , ite , jts , jte , kts , kte
END DO
. . .
END SUBROUTINE
.
,
,
)
Controlling parallelism at runtime
• Shared memory number of tiles
– 1 tile if not compiled for OpenMP or if there is only one thread
available
– unless: numtiles > 1 in namelist
– unless: tile size is specified in namelist as tile_sz_x > 0 and
tile_size_y > 0
– The tiling dimension is specified at compile time in
frame/module_machine.F as either TILE_Y, TILE_X, or TILE_XY
(this is overridden by tile_sz_x and tile_sz_y)
Controlling parallelism at runtime
• Distributed memory patches
– The number of patches is always the same as the number of MPI
processes, which WRF learns by interrogating the particular comm
package (e.g. RSL, which returns the value of MPI_Comm_size)
– The patching algorithm is built in and choses a decomposition that
is closest to square (should be more controllable)
– unless: numtiles > 1 in namelist
– unless: tile size is specified in namelist as tile_sz_x > 0 and
tile_size_y > 0
– The tiling dimension is specified at compile time in
frame/module_machine.F as either TILE_Y, TILE_X, or TILE_XY
(this is overridden by tile_sz_x and tile_sz_y)
WRF model data structures
• State data
– Fields in domain data type, defined in Registry
• Decomposed 2- and 3-D arrays; dimensions correspond to physical
domain dimensions
• 4-D “scalar” arrays (e.g. moist); accessible individually or en masse
• Boundary arrays
• Misc. un-decomposed arrays and 0-dimensional variables
– Dimensioned using “memory” dimensions
– Allocated using F90 ALLOCATE
• I1 (local to solver) data
– Also defined in Registry but not in domain data type
– Dimensioned using memory dimensions
– Automatic allocation (usually on main program stack)
• I2 (local to model layer subroutine) data
– Defined only in subroutine
– Defined using “tile” dimensions
– Automatic allocation (usually on thread-local stacks)
WRF Registry
• CASE mechanism for managing complexity of WRF code:
– Data base (ASCII flat file) of information about code
• State data and attributes
– Dimensionality, time levels, core association, other metadata
– I/O dataset membership
• Configuration data
• Packages (including multiple dyncores) with data associations
• Communication definitions
– Mechanism for compile-time generation of
•
•
•
•
Data definitions, allocations
Driver/mediation layer interfaces
I/O mechanisms
Communication mechanisms
• WRF 1.2 Registry rewritten
– Added Abstract Data Types (3DVAR)
– Additional communication options (e.g. transposes)
– More general data definition semantics
Registry Data Base
•
•
ACSCII flat file: Registry/Registry
Types of entry:
–
–
–
–
–
–
–
–
–
•
Dimspec -- Describes dimensions that are used to define arrays in the model
State – Describes state variables and arrays in the domain DDT
I1 – Describes local variables and arrays in solve
Typedef -- Describes derived types that are subtypes of the domain DDT
Rconfig – Describes a configuration (e.g. namelist) variable or array
Package – Describes attributes of a package (e.g. physics)
Halo -- Describes halo update interprocessor communications
Period -- Describes communications for periodic boundary updates
Xpose -- Describes communications for parallel matrix transposes
Mechanism in tools directory; program name is “registry”; used by WRF build
procedure
Dimspec entry
•
Elements
–
–
–
–
–
–
•
Entry: The keyword “dimspec”
DimName: The name of the dimension (single character)
Order: The order of the dimension in the WRF framework (1, 2, 3, or ‘-‘)
HowDefined: specification of how the range of the dimension is defined
CoordAxis: which axis the dimension corresponds to, if any (X, Y, Z, or C)
DatName: metadata name of dimension
Example
#<Table>
dimspec
dimspec
dimspec
dimspec
<Dim>
i
j
k
l
<Order> <How defined>
<Coord-axis>
1
standard_domain
x
3
standard_domain
y
2
standard_domain
z
2
namelist=num_soil_layers z
<DatName>
west_east
south_north
bottom_top
soil_layers
State entry
•
Elements
–
–
–
–
–
–
–
–
–
–
–
•
Entry: The keyword “state”
Type: The type of the state variable or array (real, double, integer, logical, character, or derived)
Sym: The symbolic name of the variable or array
Dims: A string denoting the dimensionality of the array or a hyphen (-)
Use: A string denoting association with a solver or 4D scalar array, or a hyphen
NumTLev: An integer indicating the number of time levels (for arrays) or hypen (for variables)
Stagger: String indicating staggered dimensions of variable (X, Y, Z, or hyphen for no
staggering)
IO: String indicating whether and how the variable is subject to I/O
DName: Metadata name for the variable
Units: Metadata units of the variable
Descrip: Metadata description of the variable
Example
#
Type Sym Dims
Use
Tlev Stag IO
Dname
Descrip
# definition of a 3D, two-time level, staggered state array
state real ru
ikj
dyn_eh
2
X
irh
"RHO_U" "X WIND COMPONENT“
# definition of fields in 4D scalar array “moist”
state real qv
ikjft moist
2
irh
"QVAPOR" "Water vapor mix ratio“
state real qc
ikjft moist
2
irh
"QCLOUD" "Cloud water mixing ratio"
Rconfig entry
•
Elements
–
–
–
–
Entry: the keyword “rconfig”
Type: the type of the namelist variable (integer, real, logical – no strings yet)
Sym: the name of the namelist variable or array
How set: indicates how the variable is set: e.g. namelist or derived, and if namelist,
which block of the namelist it is set in
– Nentries: specifies the dimensionality of the namelist variable or array. If 1 (one) it
is a variable and applies domain-wide; otherwise specify max_domains (which is
an integer parameter defined in module_driver_constants.F).
– Default: the default value of the variable to be used if none is specified in the
namelist; hyphen (-) for no default
•
Example
#
rconfig
Type
Sym
integer dyn_opt
How set
Nentries
namelist,namelist_01
1
Default
1
Package Entry
•
Elements
– Entry: the keyword “package”,
– Package name: the name of the package: e.g. “kesslerscheme”
– Associated rconfig choice: the name of a rconfig variable and the value of that
variable that choses this package
– Package state vars: unused at present; specify hyphen (-)
– Associated 4D scalars: the names of 4D scalar arrays and the fields within those
arrays this package uses
•
Example
# namelist entry that controls microphysics option
rconfig
integer
mp_physics
namelist,namelist_04
# specification of microphysics options
package
passiveqv
mp_physics==0
package
kesslerscheme mp_physics==1
package
linscheme
mp_physics==2
package
ncepcloud3
mp_physics==3
package
ncepcloud5
mp_physics==4
-
max_domains
moist:qv
moist:qv,qc,qr
moist:qv,qc,qr,qi,qs,qg
moist:qv,qc,qr
moist:qv,qc,qr,qi,qs
0
Comm entries: halo and period
•
Elements
– Entry: keywords “halo” or “period”
– Commname: name of comm operation
– Description: defines the halo or period operation
• For halo: npts:f1,f2,...[;npts:f1,f2,...]*
• For period: width:f1,f2,...[;width:f1,f2,...]*
•
Example
# first exchange in eh solver
halo
HALO_EH_A 24:u_2,v_2,ru_1,ru_2,rv_1,rv_2,w_2,t_2;4:pp,pip
# a periodic boundary update
period PERIOD_EH_A 2:u_1,u_2,ru_1,ru_2,v_1,v_2,rv_1,rv_2,rw_1,rw_2
Additional Information
•
•
•
•
[email protected]
www.wrf-model.org
WRF Design and Implementation (draft)
Tomorrow:
– How to make changes in WRF code