Transcript Diapositive 1

The European PRISM Initiative
&
the OASIS coupler
S. Valcke, CERFACS (France)
R. Budich, MPI –M (Germany)
M. Carter, UK MetOffice (UK)
M.-A. Foujols, IPSL (France)
E. Guilyardi, CNRS (France)
M. Lautenschalger, MPI-M&D (Germany)
R. Redler, NEC-CCRLE (Germany)
L. Steenman-Clark, CGAM (UK)
N. Wedi, ECMWF
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Outline
•PRISM:
•goals & benefits
•FP5 project and the Support Initiative
•organisation
•the PRISM Areas of Expertise
•OASIS:
•historical background
•the community today
•some key notes
•The OASIS4 coupler:
•model adaptation
•component model description
•coupled model configuration
•communication
•regridding/transformations
The Australian Community Climate and Earth System Simulator (ACCESS)
•grids
supported
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 2
PRISM: the goals
 Increase what Earth system modellers have in common today
(compilers, message passing libraries, algebra libraries, etc.)
Metadata
Source management
Compiling + running env.
tomorrow
Coupling & I/O
Science
today
MPI – LAPACK - …
Fortran & C Compilers
tomorrow
 Share the development, maintenance and support of a wider set of
Earth System Modelling software tools and standards.
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 3
PRISM: the benefits
 reduce the technical development efforts of each
individual research team
 facilitate assembling, running, monitoring, and
post-processing of ESMs based on state-of-the-art
component models
Help climate modellers spend more time on science:

promote the key scientific diversity

increase scientific collaboration
stimulate computer manufacturer contribution (tool
portability, optimization of next generation of platforms
for ESM needs)

The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 4
PRISM: FP5 project and Support Initiative
• 2001-2004: the PRISM EU project
– a European project funded for 4.8 M€ by the EC
– 22 partners
• 2005-2008: the PRISM Support Initiative:
– 7 partners: CERFACS, CGAM, CNRS, ECMWF, MPI-M&D, UK
MetOffice, NEC-CCRLE
– 9 associate partners: CSC, CRAY, IPSL, Météo-France, MPI-M,
NEC-HPCE, SGI, SMHI, SUN
 8 py/y for 3 years
http://prism.enes.org
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 5
PRISM: the organisation
PRISM SB Chair
P. Bougeault (ECMWF)
PRISM Steering Board
CERFACS, CGAM, CNRS, ECMWF, MPI-M&D, UK MetOffice, NEC-CCRLE
PUG chair
PRISM Coordinator(s)
E. Guilyardi (CNRS), S. Valcke (CERFACS)
R. Budich (MPI)
PRISM User
Group
PRISM Core Group
(7 people)
PRISM areas of expertise (PAEs)
Code Coupling & I/O
Integration & modelling
environments
Data processing, visualisation
and management
Meta data
Computing
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 6
PRISM: the Areas of Expertise
PRISM is organised around 5 “PRISM Areas of Expertise”:
• Promotion and, if needed, development of software tools for ESM
• Organisation of related network of experts
• Technology watch
• Promotion of community standards
• Coordination with other international efforts
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 7
PAE Code Coupling and IO
Leader: S. Valcke (CERFACS)
development and support of tools for coupling climate modelling
component codes:

• OASIS3 and OASIS4 couplers

technology watch on coupling tools developed outside PRISM:
• PALM coupler (CERFACS)
• Bespoke Framework Generator (U. of Manchester)
• CCSM (NCAR), …

relations with projects involving code coupling:
•UK Met Office FLUME project
•US ESMF project, GENIE project
•ACCESS
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 8
PAE Integration and modelling environments
Leader: M. Carter (UK MetOffice)
 source version control for software development
 code extraction and compilation

job configuration & running
• Subversion for source version control (PRISM SVN server in Hamburg)
• Standard Compiling and Running Environments (SCE/SRE, MPI-M&D):
integrated environment to compile and run different coupled models based on
different component models (standard directory structure, coding rules)
• Flexible Configuration Management (FCM, UK Met Office):
version control, compilation
• prepIFS (ECMWF):
tailored graphical interface for model configuration (Web services tech.)
 prepOASIS4: GUI to configure a coupled model using OASIS4
• Supervisor Monitor Scheduler (SMS, ECMWF):
management of networks of jobs across a number of platforms
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 9
PAE Data processing, visualisation and management
Leader: M. Lautenschlager (MPI-M&D)


•
•
data processing, visualisation, archiving and exchange for Earth
system research
standards and infrastructure for networking between
geographically distributed archives
CDO (Climate data Operators form MPI-M)
CDAT (Climate Data Analysis Tools, from PCMDI)
•
CERA-2 data model (World Climate Data Centre):
geo-referenced climate data detection, browse and use
•
MARS (ECMWF):
meteorological data access and manipulation (for major NWP sites)
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 10
PAE Meta-data
Leader: L. Steenman-Clark (CGAM)
Meta-data: data about data, models, runs, ...
… a hot topic in the last few years



exchange and use of data
interchangeability of Earth system models or modelling components
forum to discuss, develop, and coordinate metadata issues:
• Numerical Model Metadata (U. of Reading):
:
numerical code bases, simulations
•
CURATOR project (USA)
data, codes, simulations
•
Numerical grid metadata (GFDL, USA):
grid
•
netCDF CF convention (PCMDI and BADC):
climate and forecast data files
•
OASIS4 metadata
coupling and IO interface
•
UK Met Office FLUME project:
management of model configuration
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 11
PAE Computing
Leader: M.-A. Foujols (IPSL), R. Redler (NEC-CCRLE)

Computing aspects are highly important for Earth system modelling
Computer vendors have to be kept informed about requirements emerging from
the climate modelling community

Earth system modellers have to be informed about computing issues to preview
difficulties and evolutions

•
file IO and data storage
•
algorithmic development
•
portable software to fit the needs of parallel and vector systems
•
sharing of experience (e.g. work on the Earth Simulator)
•
establishment of links with computing projects (e.g. DEISA)
•
information about important conferences and workshops.
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 12
OASIS: historical background
OASIS: developed since 1991 to couple existing GCMs
1991
|--
OASIS 1  OASIS 2


2001
|--- PRISM 
OASIS3
OASIS4 
OASIS1, OASIS2, OASIS3:
•low resolution, low number of 2D fields, low coupling frequency:
flexibility
very important, efficiency not so much!
OASIS4:
•higher resolution parallel models, massively parallel platforms, 3D fields
need
to optimise and parallelise the coupler
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 13
OASIS: the community today
•CERFACS (France)
ARPEGE3 - ORCA2-LIM
ARPEGE4 - NEMO-LIM - TRIP
•METEO-FRANCE (France)
ARPEGE4 - ORCA2
ARPEGE3 - OPA8.1-GELATO
•IPSL- LODYC, LMD, LSCE (France)
LMDz - ORCA2LIM
•MERCATOR (France)
ARPEGE medias - OPAmed
LMDz - ORCA4
(for interpolation only)
•MPI - M&D (Germany)
ECHAM5 - MPI-OM
PUMA - C-HOPE
ECHAM5 - E-HOPE
ECHAM5 - C-HOPE
EMAD - E-HOPE
ECHAM4 - E-HOPE
•ECMWF
IFS - CTM
IFS - ORCA2
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 14
OASIS: the community today
•IFM-GEOMAR (Germany)
ECHAM5 - NEMO (OPA9-LIM)
•NCAS / U. Reading (UK)
ECHAM4 - ORCA2
•SMHI (Sweden)
RCA(region.) – RCO(region.)
•NERSC (Norway)
ARPEGE - MICOM
•KNMI (Netherlands)
ECHAM5 - TM5/MPI-OM
•INGV (Italy)
ECHAM5 – MPI-OM
•ENEA (Italy)
MITgcm - REGgcm
•JAMSTEC (Japan)
ECHAM5(T106) - ORCA ½ deg
•IAP-CAS (China)
AGCM - LSM
•BMRC (Australia)
TCLAPS - MOM
•U. of Tasmania (Australia)
Sea Ice code - MOM4
•RPN-Environment Canada (Canada)
MEC - GOM
•UQAM (Canada)
GEM - RCO
•U. Mississippi (USA)
MM5 - HYCOM
•IRI (USA)
ECHAM5 - MOM3
•JPL (USA)
UCLA-QTCM The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
HADAM3-ORCA2
Trident-Ind4-Atlantic
Slide 15
OASIS: Some key notes

Developers: CERFACS, NEC CCRLE, CNRS, SGI, (NEC HPCE)

Public domain; open source license (LGPL)

Programming language: Fortran 90 and C

Public domain libraries; vendor optimized versions may exist:
• MPI1 and/or MPI2; NetCDF/parallel NetCDF; libXML
• mpp_io; SCRIP

Static coupling
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 16
The OASIS3 coupler
• Coupler developed since more than 15 years in CERFACS
• Stable, well-debugged, but limited
• Last version: oasis3_prism_2-5 delivered in September 2006
• User support provided but most development efforts go to OASIS4
• Mono-process coupler + parallel coupling library (PSMILe)
• synchronisation of the component models
• coupling fields exchange
• I/O actions
• mono-process interpolation
• Platforms:
•Fujitsu VPP5000, NEC SX5-6-8, Linux PC, IBM Power4, CRAY XD1,
Compaq, SGI Origin, SGI O3400
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 17
OASIS3: model adaptation
PRISM System Model Interface Library (PSMILe) API :
•Initialization
•Global grid definition (master process only)
•Local partition definition
•Coupling or I/O field declaration
•Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in namcouple:
•user’s defined source or target (end-point communication)
•coupling or I/O sending or receiving at appropriate times
•automatic averaging/accumulation
•automatic writing of coupling restart file at end of run
call prism_put
call prism_get
(var_id, time, var_array, ierr)
(var_id, time, var_array, ierr)
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 18
OASIS3: coupled model configuration
•In text file namcouple, read by OASIS3 main process, and
distributed to component model PSMILes at beginning of run:
•total run time
•component models
•number of coupling fields
•for each coupling field:
•source and target names (end-point communication) (var_name)
•grid acronym (grid_name)
•coupling and/or I/O status
•coupling or I/O period
•transformations/interpolations
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 19
OASIS3: communication
PSMILe based on MPI1 or MPI2 message passing
O
•Parallel communication between parallel models
and Oasis3 interpolation process
A
O
A
Oasis3
O
A
O
A
B
•Direct communication between models with
same grid and partitioning
A
B
A
B
•I/O functionality (automatic switch between
A
coupled and forced mode): GFDL mpp_io library
A
file
A
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 20
OASIS3: interpolations/transformations
separate sequential process
Oasis3
neighbourhood search
weight calculation
interpolation per se during the run
O
A
A
O
Oasis3
A
O
O
on 2D scalar or vector fields
• Interfacing with RPN Fast Scalar INTerpolator package
• nearest-neighbour, bilinear, bicubic for regular Lat-Lon grids
• Interfacing with SCRIP1.4 library
•nearest-neighbour, 1st and 2nd order conservative remapping for all grids
• bilinear and bicubic interpolation for «logically-rectangular» grids
• Bilinear and bicubic interpolation for reduced atmospheric grids
• Other spatial transformations: flux correction, merging, etc.
• General algebraic operations
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 21
The OASIS4 coupler
•A parallel central Driver/Transformer:
•launches models at the beginning of the run (MPI2)
•reads the user-defined configuration information and
distributes it to the component PSMILes
•performs parallel transformations of the coupling fields during
the run
•A parallel model interface library (PSMILe) that performs:
•weight-and-address calculation for the coupling field
interpolations
•MPI-based coupling exchanges between components
•component I/O (GFDL mpp_io library)
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 22
OASIS4: model adaptation (1/3)
•Initialization:
call prism_init_comp
(comp_id, comp_name, ierr)
call prism_get_localcomm
(comp_id, local_comm, ierr)
•Definition of grid (3D)
call prism_def_grid
(grd_id, grd_name, comp_id, grd_shape, type, ierr)
call prism_set_corners(grd_id,
nbr_crnr, crnr_shape, crnr_array, ierr)
•Placement of scalar points and mask on the grid:
call prism_set_points (pt_id,
pt_name, grd_id, pt_shape,
pt_lon, pt_lat, pt_vert ,ierr)
call prism_set_mask
(msk_id, grd_id, msk_shape, msk_array, ierr)
Function overloading to keep the interface concise and flexible
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 23
OASIS4: model adaptation (2/3)
prism_def_grid
prism-set-corners
prism_set_points
prism_set_points
prism_set_points
prism_set_mask
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 24
OASIS4: model adaptation
(3/3)
•Coupling or I/O field declaration
call prism_def_var
(var_id, var_name, grd_id, pt_id, msk_id, var_nbrdims,
var_shape, var_type, ierr)
•End of definition
call prism_enddef
(ierr)
•Coupling or I/O field sending and receiving:
in model time stepping loop
depending on user’s specifications in SMIOC:
•user’s defined source or target, component or file (end-point communication)
•coupling or I/O sending or receiving at appropriate times
•averaging/accumulation
call prism_put
call prism_get
(var_id, date, date_bounds, var_array, info, ierr)
(var_id, date, date_bounds, var_array, info, ierr)
•Termination
call prism_terminate
(ierr)
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 25
OASIS4: component model description
Application and component description (XML files):
For each application (code): one Application Description (AD):
•possible number of processes
•components included
For each component in the application:
one Potential Model Input and Output Description (PMIOD)
•component general characteristics: name, component simulated, …
•grid information: domain, resolution(s), grid type, …
•potential I/O or coupling variables:
•local name, standard name
•units, valid min and max
•numerical type
•associated grid and points
•intent –input and/or output
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 26
OASIS4: coupled model configuration
(Through a GUI), the user produces
a Specific Coupling Configuration (SCC):
• experiment and run start date and end date
• applications, components for each application
• host(s), number of processes per host, ranks for each component
For each component,
a Specific Model Input and Output Configuration (SMIOC)
• grid information: chosen resolution, …
• I/O or coupling variables:
•name, units, valid min max, numerical type, grid
•activated intent –input and/or output
•source and/or target (component and/or file)
•coupling or I/O dates
•transformations/interpolations/combinations
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 27
ATM-LAND AD
Driver
user
ATM PMIOD
Va1: in, metadata Va1
Va2: out, metadata Va2
Va3: out, metadata Va3
user
OCE AD
OCE PMIOD
Vo1: out, metadata Vo1
Vo2: in, metadata Vo2
SCC
ATM:...
OCE:...
LAND:...
user
user
OCE SMIOC
Vo1 : to ATM Va1, T1
to File1
Vo2 : from ATM Vo1
ATM
OCE
ATM SMIOC
Va1 : from OCE Vo1
Va2: to OCE Vo2, T2
Va3 : to LAND Vl1
Va2
Vo2
T
Va1
LAND
Vo1
Vl2
Va3
Vl3
LAND PMIOD
Vl1: in, metadata Vl1
Vl2: in, metadata Vl2
File1
Definition Phase
File2
Composition Phase
user
LAND SMIOC
Vl1 : from ATM Va3
Vl2 : from file File2
Deployment Phase
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 28
OASIS4 communication (1/2)
•Model interface library: PSMILe based on MPI1 or MPI2
•Parallel communication including repartitioning:
•based on geographical description of the partitions
•parallel calculation of communication patterns in source PSMILe
OB C
OB C
C
O1
C
O1
C
T
C O2
O2
OB C
OB C
OB C
T
Same grid, different decomposition
 direct repartitioning:
• for each target point, parallel
calculation of source matching point in
source PSMILe
OB C
Different grid and decomposition
 via parallel Transformer:
• for each target point, parallel calculation
of source interpolation neighbours in
source PSMILe
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 29
OASIS4: communication (2/2)
• end-point communication (source doesn’t know target and vice-versa)
• parallel 3D neighbourhood search, based on efficient multigrid
algorithm, in each source process PSMILe
• extraction of useful part of source field only
• one-to-one, one-to-many
• parallel I/O (vector, bundles, vector bundles) : GFDL mpp_io, parNetCDF
A
A
A
single
file
A
file1
A
A
file2
A
A
file3
A
distributed
file
parallel
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 30
OASIS4 regridding/transformations
• source time transformations (prism_put):
•average, accumulation
• target time transformations (prism_get)
•time interpolation (for I/O only)
• statistics
• local transformations:
• addition/multiplication by scalar
• interpolation/regridding (3D):
• nearest-neighbour 2D in the horizontal, “none” in the vertical
•
•
•
•
nearest-neighbour 3D
bilinear in the horizontal, “none” in the vertical
bicubic (gradient, 16 nghbrs) in the horizontal, “none” in the vertical
trilinear
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 31
Grids supported by OASIS4
• Regridding, repartitioning, I/O:
• Regular in lon, lat, vert (“Reglonlatvrt”):
• lon(i), lat(j), height(k)
• Irregular in lon and lat, regular in the vert
(“irrlonlat_regvrt”):
• lon(i,j), lat(i,j), height(k)
• Irregular in lon, lat, and vert
(“irrlonlatvrt”) (not fully tested)
• lon(i,j,k), lat(i,j,k), height(i,j,k)
• Gaussian Reduced in lon and lat, regular in the vert
(“Gaussreduced_regvrt”)
• lon(nbr_pt_hor), lat(nbr_pt_hor), height(k)
•
Repartitioning and I/O only:
• “Non-geographical” fields
• no geographical information attached
• local partitions described in the global index space (prism_def_partition)
•I/O only:
• Unstructured grids (“unstructlonlatvrt”)
•lon(npt_tot), lat(npt_tot), height(npt_tot)
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 32
OASIS4: developments & perspectives
•OASIS4 tested and run with toy examples on:
•NEC SX6 and SX8
•IBM Power4
•SGI O3000/2000
•SGI IA64 Linux server Altix 3000/4000
•Intel® Xeon™ Infiniband and Myrinet Clusters
•Linux PC DELL
•OASIS4 now being used in a reduced number of real applications:
•EU project GEMS: atmospheric dynamic and chemistry coupling
•SMHI: ocean-atmosphere regional coupling
•UK Met Office: global ocean-atmosphere coupling (currently prototyping)
•IFM-GEOMAR (Kiel) in pseudo-models to interpolate high-resolution fields.
•Current developments:
•
•
•
•
2D conservative remapping
Parallel global search for the interpolation
Transformer efficiency
Full validation of current transformations
•Full public release planned beginning 2007.
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 33
Conclusions
PRISM provides:
 framework promoting common software tools for Earth system modelling
 some (more or less) standard tools (OASIS, source management, compiling, …)
 network allowing ESM developers to share expertise and ideas
 visible entry point for international coordination
 metadata definition
 WCRP white paper with ESMF on “Common Modeling Infrastructure for the
International Community”

PRISM current decentralized organisation (bottom-up approach):
 allows “best of breed” tools to naturally emerge
 relies on the developments done in the different partner groups

Additional funding needed:
• more networking and coordination activities
• specific technical developments within the partner groups

Additional contributors are most welcome to join !
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 34
The end
The Australian Community Climate and Earth System Simulator (ACCESS)
- Challenges & Opportunities, Melbourne, 28/11-01/12 2006
Slide 35