Transcript Document

OASIS3 and OASIS4 : the PRISM couplers
FLUME-PRISM Workshop
Met Office, Exeter, March 15-16, 2004
G. Berti, P. Bourcier, A. Caubel, D. Declat, M.-A. Foujols, J. Latour, S. Legutke,
J. Polcher, R. Redler, H. Ritzdorf, T. Schoenemeyer, S. Valcke and R. Vogelsang
Outline
OASIS historic
OASIS community today
PRISM first coupler: Oasis3
•model adaptation
•coupled model configuration
•communication
•interpolations/transformations
PRISM final coupler: Oasis4
•model adaptation
•the Driver
•communication
•interpolations/transformations
•in the prototype
•still to be done
Conclusions
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 2
OASIS historic
OASIS: Ocean Atmosphere Sea Ice Soil
Developed since 1991 in CERFACS to couple existing GCMs.
At the time:
•Models at relatively low resolution (~10000-20000 pts)
•Small number of 2D coupling fields (~10)
•Low coupling frequency (~once/day)
flexibility was very important, efficiency not so much!
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 3
OASIS historic
1993: OASIS 1.0 (CRAY pipes & files)
1995: OASIS 2.0 (PVM3): Heterogeneous coupling (Cathode)
1996: OASIS 2.1: Global warning scenario (SIDDACLICH)
1998: OASIS 2.2 (SIPC): Seasonal forecasting (Demeter)
1999: OASIS 2.3 (GMEM - NEC): Intercomparison (SINTEX)
2000: OASIS 2.4 (MPI2): Climate Natural variability (Predicate)
2003: OASIS 3 (MPI2-MPI1 + PSMILe + I/O): PRISM
=> new Oasis3 (oasis3_prism_2-1) now available !
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 4
OASIS community today
•CERFACS (France)
ARPEGE3
-
OPA 8.1
•METEO-FRANCE (France)
ARPEGE4
ARPEGE3
-
ORCA2
OPA8.1
ARPEGE3
ARPEGE3
-
ARPEGE medias -OPAmed
ARPEGE2
- OPA TDH
•IPSL- LODYC, LMD, LSCE (France) LMDz
LMDz
IFS
-
ORCA4
OPA 8.1
-
LMDz
ECHAM4
•MERCATOR (France)
ORCA2LIM
OPA ATL3/ATL1
ORCA2
PAM(OPA)
•MPI - M&D (Germany)
ECHAM5
PUMA
ECHAM5
ORCA2LIM
OPAICE
-
MPI-OM
C-HOPE
E-HOPE
ECHAM5
EMAD
ECHAM4
-
IFS Cy23r4 -
E-HOPE
IFS Cy15r8 -
C-HOPE
E-HOPE
E-HOPE
•ECMWF (UK)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
E-HOPE
Slide 5
OASIS community today
•CGAM-Reading (UK)
HadAM3
-
ORCA2
•Southampton University (UK)
Inter. Atm
-
OCCAM lite
•UCL (Belgium)
LMDz
•SMHI (Sweden)
RCA(region.) –
RCO(region.)
•U. of Bergen (Norway)
MM5
-
ROMS
•KNMI (Netherlands)
ECHAM5
-
MPI-OM
•DMI (Danemark)
ECHAM
-
HIRLAM
•INGV (Italy)
ECHAM5
–
MPI-OM
•IRI (USA)
ECHAM4
-
MOM3
•JAMSTEC (Japan)
ECHAM4
-
OPA 8.2
BAM3
-
ACOM2
•U. of Tasmania (Australia)
Data Atm.
-
MOM4
•CAS,IIT Delhi (India)
MM5
-
POM
•BMRC (Australia)
ECHAM - RCA
- CLIO
BAM - MOM4
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 6
PRISM first coupler: Oasis3
Mono-process coupler + parallel coupling library (PSMILe)
• synchronisation of the component models
• coupling fields exchange
• I/O actions
• mono-process interpolation
Flexibility, modularity:
•
•
•
Arbitrary number of models and coupling fields
Different coupling frequencies for each field
Different transformations for each field
Static coupling: all coupling parameters defined initially.
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 7
Oasis3: model adaptation
(1/3)
PRISM System Model Interface Library (PSMILe) API :
•Initialization:
call prism_init_comp
(comp_id, comp_name, ierr)
•Retrieval of component local communicator:
call prism_get_localcomm
•Global grid definition
call
call
call
call
(local_comm, ierr)
(master process only)
prism_write_grid (grd_name, nx, ny, lon, lat)
prism_write_corner (grd_name, nx, ny, nc, clon,
prism_write_mask (grd_name, nx, ny, mask)
prism_write_areas (grd_name, nx, ny, area)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
clat)
Slide 8
Oasis3: model adaptation
(2/3)
PRISM System Model Interface Library (PSMILe) API :
•Local partition definition
(by each process in the global index space)
call prism_def_partition
(part_id, ig_paral, ierr)
•Coupling or I/O field declaration (name, partition id, status, shape, type)
call prism_def_var
(var_id, var_name, part_id, var_nbrdims, var_inout,
var_shape, var_type, ierr)
•End of definition
call prism_enddef
(ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 9
Oasis3: model adaptation
(3/3)
PRISM System Model Interface Library (PSMILe) API: :
•Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in namcouple:
•user’s defined source or target (end-point communication)
•coupling or I/O sending or receiving at appropriate times
•automatic averaging/accumulation
•automatic writing of coupling restart file at end of run
call prism_put
call prism_get
(var_id, time, var_array, ierr)
(var_id, time, var_array, ierr)
•Termination:
call prism_terminate
(ierr)
•Auxiliary routines:
call prism_put_inquire
call prism_put_restart
(var_id, time, info)
(var_id, time, info)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 10
Oasis3: coupled model configuration
•In text file namcouple, read by Oasis3 main process, and
distributed to component model PSMILes at beginning of run:
•total run time
•component models
•number of coupling fields
•for each coupling field:
•source and target names (end-point communication) (var_name)
•grid acronym (grid_name)
•coupling and/or I/O status
•coupling or I/O period
•transformations/interpolations
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 11
Oasis3: communication
PSMILe based on MPI1 or MPI2 message passing
•End-point communication; one-to-one
O
A
O
A
•Parallel communication between parallel models
and Oasis3 interpolation process
•Direct communication between models with
same grid and partitioning
•I/O functionality (automatic switch between
coupled and forced mode): GFDL mpp_io library
Oasis3
O
A
O
A
B
A
B
A
B
A
A
file
A
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 12
Oasis3: interpolations/transformations
separate sequential process
O
Oasis3
neighbourhood search
weight calculation
interpolation per se during the run
A
A
O
Oasis3
O
A
O
on 2D scalar fields only
• Interfacing with RPN Fast Scalar INTerpolator package
• nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids
• Interfacing with SCRIP1.4 library
•nearest-neighbour, 1st and 2nd order conservative remapping for all grids
• bilinear and bicubic interpolation for «logically-rectangular» grids
• Bilinear and bicubic interpolation for reduced atmospheric grids
• Other spatial transformations: flux correction, merging, etc.
• General algebraic operations
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 13
PRISM final coupler: Oasis4
Situation today:
• Higher resolution, parallel and scalable models
• Higher coupling frequencies desirable
• Higher number of models and (3D) coupling fields
• Massively parallel platforms
Need to optimise and parallelise the coupler
Oasis4 is composed of:
• a Driver
• a Transformer
• a new PRISM System Model Interface Library
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 14
PRISM final coupler: Oasis4
Some keynotes:
Developers:
Sophie Valcke (CERFACS)
Damien Declat (CERFACS)
René Redler (NEC CCRLE)
Hubert Ritzdorf (NEC CCRLE)
Reiner Vogelsang (SGI)
Th. Schoenemeyer (NEC HPCE)
Jean Latour (Fujitsu )
- Design, XML input
- Driver, transformer, XML input
- PSMIle, portability to Linux PC cluster
- Parallel search, transformer interaction
- I/O, portability to SGI
- Parallelisation, portability to NEC SX
- Portability to Fujitsu VPP and PC cluster
Public domain; open source license (LGPL)
All external libraries and modules used exist in public domain versions;
vendor optimized versions may also exist:
•
•
•
•
MPI1 and/or MPI2
mpp_io; NetCDF/parallel NetCDF
libXML
SCRIP
Programming language: Fortran 90 and C
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 15
Oasis4: model adaptation
(1/5)
New PSMILe API:
•Application initialization (1 call/application) :
call prism_init
(appl_name, ierr)
•Component application (1 call/component) :
call prism_init_comp
(comp_id, comp_name, ierr)
•Retrieval of component local communicator (1 call/component):
call prism_get_localcomm
(comp_id, local_comm, ierr)
•Inquire SCC or SMIOCs persistent values:
call prism_get_persist (…)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 16
Oasis4: model adaptation
(2/5)
•Definition of grid (3D)
call prism_def_grid
(grd_id, grd_name, comp_id, grd_shape, type, ierr)
call prism_set_corners
(grd_id, nbr_crnr, crnr_shape, crnr_array, ierr)
•Placement of scalar points, vector points, subgrids on the grid:
call prism_set_points
(pt_id, pt_name, grd_id, pt_shape, pt_lon, pt_lat, pt_vert ,ierr)
call prism_set_vector
(vct_id, vct_name, array_of_pt_ids,ierr)
call prism_set_subgrid
(sgrd_id, sgrd_name, grd_id, n_sgrd, sgrd_shape, sgrd_array, ierr)
Function overloading to keep the interface concise and flexible
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 17
Oasis4: model adaptation
(3/5)
•Definition of grid cell dimension
call prism_set_scalefactors
(grd_id, scale_shape, scale_array, ierr)
•Definition of grid cell mask
call prism_set_mask
(msk_id, grd_id, msk_shape, msk_array, ierr)
•Definition of point local coordinate system angle
call prism_set_angle
(pt_id, angle_shape, angle_array, ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 18
Oasis4: model adaptation
(4/5)
prism_def_grid
prism-set-corners
prism_set_points
prism_set_points
prism_set_points
prism_set_mask
prism_set_vector
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 19
Oasis4: model adaptation
(5/5)
•Coupling or I/O field declaration
call prism_def_var
(var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims,
var_shape, var_type, ierr)
•End of definition
call prism_enddef
(ierr)
•Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in SMIOC:
•user’s defined source or target, component or file (end-point communication)
•coupling or I/O sending or receiving at appropriate times
•automatic averaging/accumulation
•automatic writing of coupling restart file at end of run
call prism_put
call prism_get
(var_id, date, date_bounds, var_array, info, ierr)
(var_id, date, date_bounds, var_array, info, ierr)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 20
Oasis4: coupled model configuration
(1/2)
Application and component description (XML files):
For each application (code):
one Application Description (AD):
•possible number of processes
•components included
For each component in the application:
one Potential Model Input and Output Description (PMIOD)
•component general characteristics: name, component simulated, …
•code general characteristics: lab, documentation, contact, …
•grid information: domain, resolution(s), grid type, …
•potential I/O or coupling variables:
•local name, standard name
•units, valid min and max
•numerical type
•associated grid and points
•intent –input and/or output
•persistent variables: name, units, numerical type, possible values, …
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 21
Oasis4: coupled model configuration
(2/2)
Coupled model configuration (XML files): The user produces
a Specific Coupling Configuration (SCC):
• experiment and run start date and end date
• start mode (MPI1, MPI2)
• applications, components for each application
• host(s), number of processes per host, ranks for each component
For each component,
a Specific Model Input and Output Configuration (SMIOC)
• PMIOD component and code general characteristics
• grid information: chosen resolution, …
• I/O or coupling variables:
•local and standard name, units, valid min max, numerical type, grid
•activated intent –input and/or output
•source and/or target (component and/or file)
•coupling or I/O dates
•transformations/interpolations/combinations
•persistent variables: name, units, numerical type, chosen value, …
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 22
Driver
OCE AD
ATM PMIOD
V1: in, metadata V1
V2: out, metadata V2
V3: out, metadata V3
user
OCE PMIOD
V1: out, metadata V1
V2: in, metadata V2
SCC
ATM:...
OCE:...
LAND:...
user
user
OCE SMIOC
V1 : to ATM, T1
to file V1
V2 : from ATM, T2
ATM
OCE
ATM SMIOC
V1 : from OCE, T1
V2: to OCE, T2
V3 : to LAND
V2
V2
V1
ATM-LAND AD
user
T
V3
V1
LAND
V4
LAND PMIOD
V3: in, metadata V3
V4: in, metadata V4
fileV1
Definition Phase
Composition Phase
Deployment Phase
fileV4
user
LAND SMIOC
V3 : from ATM
V4 : from fileV4
Oasis4: the Driver
•Manages a static coupling:
Driver
all applications are started directly in the run
script (MPI1 mode) or launched by the Driver at
the beginning of the run (MPI2 mode) for the
whole run.
•Participates to communicator set-up
•Reads the SCC and SMIOcs information and
distributes it to the component PSMILes
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 24
Oasis4: communication
(1/3)
•More elaborate PSMILe based on MPI1 or MPI2
•Parallel communication including repartitioning:
•based on geographical description of the partitions
•parallel calculation of communication patterns in source PSMILe
OB C
OB C
OB C
OB C
C
O1
C
O1
OB C
Same grid, different decomposition
 direct repartitioning:
• for each target point, parallel
calculation of source matching point
in source PSMILe
OB C
T
C
O2
C O2
Different grid and decomposition
 via Transformer:
• for each target point, parallel
calculation of source interpolation
neighbours in source PSMILe
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 25
Oasis4: communication
Parallel calculation of communication
patterns in source PSMILe
For each pair of source and target processes:
1/ Envelop exchange
2/ Detailed neighbourhood search
xxx
xox o
x o
xox xo o
xx
o o ox x
o o ox x
xxxxx
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
(2/3)
xxxxx
xxxxx
xxxxx
xxxxx
o o o
o o o
o o o
3/ EPIOS and EPIOT definition
xxx
xxx
EPIOS1
xxx
o o o
EPIOT1
o o o
xx
xx
o o o
EPIOS2
EPIOT2
xx
o o o
xxxxx
Slide 26
Oasis4: communication
(3/3)
• parallel 3D neighbourhood search in each source process PSMILe
• new search algorithm: parallel multigrid technique, higher efficiency
• extraction of useful part of source field only
• end-point communication
• one-to-one or one-to-many; combination
• source data can be consumed by more than one target model or file
and at a different frequencies
• parallel I/O (vector, bundles, vector bundles) : GFDL mpp_io, parNetCDF
A
A
A
single
file
A
file1
A
A
file2
A
A
file3
A
parallel
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
file
distributed
Slide 27
Oasis4: interpolations/transformations
PSMILe:
•time transformations, local transformations
•global conservation, combinations
Transformer:
•loops over PSMILe requests
OB C
OB C
C
T
O2
C O2
OB C
•EPIOS and EPIOT definition (source PSMILe)
•For each EPIOT point, neighbours in EPIOS (source PSMILe)
=>Transformer calculates interpolation weights
•Prism_put of EPIOS (source PSMILe)
=>Transformer performs interpolation per se
•Prism_get of EPIOT (target PSMILe)
•well suited for MPI parallelisation
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 28
Oasis4: interpolations/transformations
Different interpolation schemes:
•nearest-neighbour, bilinear, bicubic, conservative remapping, …
On different types of fields
•1D-2D-3D scalar, vector, bundles, bundles of vectors
On different types of grids
•blockstructured (regular and irregular)
•unstructured
•static or dynamic
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 29
Oasis4: in the prototype
(03/2004)
•Access of XML information
•Coupling and I/O of n parallel applications with m components.
•Coupling exchange with repartitionning, direct or through the
(mono-process) Transformer
•Interpolations:
•PSMILe (non-exact) parallel neighborhood search
•3D nearest-neighbour
•I/O:
•single and parallel mode
•Coupling and I/O exchange from one source to many targets
•Local transformations (scatter, gather, add or mult scalar, statistics)
•Basic time transformation (average, accumul, min, max)
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 30
Oasis4: still to be done
•Completion of PSMILe calendar routines (internal and external)
•PSMILe API for model access to SCC and SMIOC information
•Interpolation:
•More schemes (conservative, 3D, etc.)
•Exact parallel neighbour search
•Transformer parallelisation
•Field reduction, combination
•Full support of vector and bundle fields (I/O OK)
•I/O: distributed mode (parNetCDF)
•Adaptative grids
•Unstructured grids
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 31
Conclusions
Calendar:
•Oasis3_prism_2-1 available
•Oasis4 prototype available within days
•Oasis4 final PRISM version due 12/2004
Follow-on:
•Oasis4 developments will go on after 12/2004 with PRISM sustained
support
•FP6 funding (march 2005)?
Static vs dynamic couplers:
•In the near and mid future, Oasis4 and Palm (Cerfacs dynamical
coupler) will remain separate software
•Fusion may be considered when climate modelling and data
assimilation communities will join
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 32
The end
FLUME-PRISM workshop – Met Office, Exeter, 15-16, 2004
Slide 33