Transcript NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH Architecture of the.

NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Architecture of the Earth System
Modeling Framework
GFDL FMS Suite
NASA GSFC PSAS
NCAR/LANL CCSM
MITgcm
NCEP Forecast
NSIPP Seasonal Forecast
V. Balaji / GFDL, Cecelia DeLuca / NCAR, Chris Hill / MIT, Max Suarez, Arlindo da Silva / NASA GSFC
EGS - AGU
Nice, France
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Technological Trends
In climate research and NWP...
increased emphasis on detailed representation of individual
physical processes; requires many teams of specialists to
contribute components to an overall modeling system.
In computing technology...
increase in hardware and software complexity in highperformance computing, as we shift toward the use of scalable
computing architectures and open community software
standards.
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Project Overview
GOAL: To increase software reuse, interoperability, ease of use and performance
portability in climate, weather, and data assimilation applications
PRODUCTS:
•
Coupling superstructure and utility infrastructure software
•
Synthetic code suite for validation and demonstration
•
Set of 15 ESMF-compliant applications (including CCSM, WRF, GFDL models;
MIT, NCEP and NASA data assimilation systems)
•
Set of 8 interoperability experiments
RESOURCES: $10.1M over 3 years from NASA Earth Science Technology Office
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Collaborators
NSF NCAR
Tim Killeen, PI
Byron Boville
Cecelia DeLuca
Roberta Johnson
John Michalakes
Al Kellie
Jeff Anderson
David Neckels
Earl Schwab
Nancy Collins
Jon Wolfe
Silverio Vasquez
Dennis Flanigan
Robbie Staufer
DOE LANL
Phil Jones
8 April 2003
NOAA NCEP
Stephen Lord
Mark Iredell
Mike Young
Weiyu Yang
John Derber
MIT
John Marshall, PI
Chris Hill
NASA DAO
Arlindo da Silva, PI
Leonid Zaslavsky
Will Sawyer
Carlos Cruz
University of Michigan
Quentin Stout
NOAA GFDL
Ants Leetmaa
V. Balaji
Robert Hallberg
Shep Smithline
NASA NSIPP
Max Suarez
Michele Rienecker
Christian Keppenne
Atanas Trayanov
DOE ANL
Rob Jacob
Jay Larson
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Interoperability
Demonstrations
COUPLED CONFIGURATION
NEW SCIENCE ENABLED
GFDL B-grid atm / MITgcm ocn
Global biogeochemistry (CO2, O2), SI timescales.
GFDL MOM4 / NCEP forecast
NCEP seasonal forecasting system.
NSIPP ocean / LANL CICE
Sea ice model for extension of SI system to centennial time scales.
NSIPP atm / DAO analysis
Assimilated initial state for SI.
DAO analysis / NCEP model
Intercomparison of systems for NASA/NOAA joint center for satellite
data assimilation.
DAO CAM-fv / NCEP analysis
Intercomparison of systems for NASA/NOAA joint center for satellite
data assimilation.
NCAR CAM Eul / MITgcm ocn
Improved climate predictive capability: climate sensitivity to large
component interchange, optimized initial conditions.
NCEP WRF / GFDL MOM4
Development of hurricane prediction capability.
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Related
Projects
PRISM
WRF
CCA
ESMF
FMS
8 April 2003
SWMF
SciDAC
ESMF Earth System Modeling Framework
CCA DOE Common Component Architecture
SciDAC DOE/NSF CCSM SciDAC Project
GEMS Goddard Earth Modeling System
FMS GFDL Flexible Modeling System
SWMF Space Weather Modeling Framework
WRF Weather Research and Forecast Model
CCSM Community Climate System Model
PRISM Program for Int. Earth System Modeling
CCSM
GEMS
Larson/ANL
DeLuca/NCAR-SCD
Jones/LANL
Stout/U Mich
Killeen/NCAR
Drake/ORNL
Boville/NCAR-CGD
Michalakes/NCAR-MMM
Suarez/NASA Goddard
Balaji/NOAA GFDL
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and PRISM
• A European Earth system modeling infrastructure
project, started December 2001
• Funded by the European Commission ($4.8M)
• Involves current state-of-the-art atmosphere,
ocean, sea-ice, atmospheric chemistry, landsurface and ocean-biogeochemistry models
• 22 partners: leading climate researchers and
computer vendors, includes MPI, KNMI, UK Met
Office, CERFACS, ECMWF, DMI.
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and PRISM, cont.
• Working together to supplement
CF convention for physical field
names and quantities
• Component interface database
developed by ESMF stores fields
in model import and export states
• V. Balaji (GFDL) is ESMF liaison to
PRISM
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF and CCA
• Common Component Architecture (CCA) is creating a minimal interface
and sets of tools for linking high performance components. CCA can
be used to implement frameworks and standards developed in specific
domains (such as ESMF).
• DOE funded through SciDAC.
• Collaborators include LANL, ANL, LLNL, ORNL, Sandia, University of
Tennessee, and many more. Ongoing ESMF collaboration with
CCA/LANL on language interoperability.
• Shujia Zhou / Arlindo da Silva (NASA GSFC) currently prototyping an
ESMF API using CCA tool CCAFFEINE; working towards CAM/DAO
PSAS coupling using ESMF and CCA.
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Architecture
Coupling Layer
ESMF Superstructure
Model Layer
User Code
Fields and Grids Layer
ESMF Infrastructure
Low Level Utilities
External Libraries
8 April 2003
BLAS, MPI, NetCDF, …
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Components
1.
ESMF provides an environment for assembling geophysical components
into an application.
Application Component
Gridded Components
2.
ESMF provides a toolkit that
components use to
i. increase interoperability
ii. improve performance
portability
iii. abstract common services
8 April 2003
Coupler Components
Component: Init( ), Run( ), Checkpoint( )
Field: Halo( ),
Import( ), Export( ) +
I/O
Grid: Regrid( ),
Transpose( ) + Metrics
Layout, PE List, Machine Model
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
General Features
• ESMF will be usable by models written in F90/C/C++
• ESMF will be usable by models requiring adjoint capability
• ESMF will be usable by models requiring shared or distributed memory
parallelism semantics
• ESMF will support SPMD and MPMD coupling
• ESMF will support several I/O formats, including GRIB/BUFR, netCDF,
HDF
• ESMF will have uniform syntax across platforms
• ESMF will target a broad range of platforms, from major centers 
commodity hardware
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Scalable Applications
Since each ESMF application is also a component, entire ESMF
applications may be treated as Gridded Components and nested
within larger applications.
climate_comp
ocn2atm_coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
PE
8 April 2003
atm_dyn
Example: atmospheric
application containing multiple
coupled components within a
larger climate application
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Local Communication
All inter-component communication within ESMF is local.
climate_comp
This means:
atm2ocn _coupler
ocn_comp
atm_comp
phys2dyn_coupler
atm_phys
PE
8 April 2003
atm_dyn
Coupler Components must be
defined on the union of the PEs of
all the components that they
couple.
In this example, in order to send
data from the ocean component to
the atmosphere, the Coupler
mediates the send.
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Modularity
Gridded Components don’t have access to the internals of other Gridded
Components. Gridded Components may
- pass their States through their argument list or
- receive methods, through their argument list, for interacting with other
Components (Transforms).
coupler
transform
call ESMF_CompRun(atm, xform)
call ESMF_CompRun(ocn, xform)
ocn_component
atm_component
call ESMF_StateXform(ex_state, &
xform)
call ESMF_StateXform(xform, &
im_state)
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Design Principles:
Uniform Communication API
The same programming interface is used for shared memory, distributed memory,
and combinations of both.
Machine model = abstraction of machine architecture (num_nodes,
num_pes_per_node)
DE = a decomposition element - may be virtual, thread or MPI process
Layout = an arrangement of DEs, in which dimensions requiring faster
communication may be specified and resources arranged accordingly
A 2 x 6 Layout of 4 3-processor nodes
The data in a Grid is decomposed
into the number of chunks specified
in the Layout.
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
ESMF Class Structure
Gridded
Component
Coupler
Component
State
Superstructure
Infrastructure
Bundle
Regrid
Field
Grid
PhysGrid
DistGrid
Layout
More work
Prototype exists
8 April 2003
MachineModel
F90
Data Communications
Array
Comm
Utilities: TimeMgr, LogErr, I/O etc.
Route
C++
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Outline
•
•
•
•
•
ESMF Project Overview
Related Projects and Interactions
ESMF Design Overview
ESMF Design Principles
Timeline and Status
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
Timeline
May 2002
Draft Developer’s Guide and Requirements Document completed;
Community Requirements Meeting and review held in D.C.
July 2002
ESMF VAlidation (EVA) suite assembled
August 2002
Architecture Document: major classes and their relationships;
Implementation Report: language strategy and programming
model; Software Build and Test Plan: sequencing and validation
May 15 2003
First API and software release, Community Meeting at GFDL
July 2003
First 3 interoperability experiments completed
April 2004
Second API and software release, Community Meeting
July 2004
All interoperability experiments completed
Jan 2005
Final release
Currently planning for ongoing funding and extensions
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
May 2003 Release
Focus for May 2003 ESMF release is on developing
sufficient infrastructure and superstructure to achieve the
initial set of interoperability experiments.
These are:
– FMS B-grid atmosphere coupled to MITgcm ocean
– CAM atmosphere coupled to NCEP analysis
– NSIPP atmosphere coupled to DAO analysis
8 April 2003
NSF NCAR / NASA GSFC / DOE LANL ANL / NOAA NCEP GFDL / MIT / U MICH
More information
2nd ESMF Community Meeting at GFDL May 15, 2003
ESMF website: http://www.esmf.ucar.edu
On the website:
• Information about the Community Meeting
• ESMF documents
• ESMF browsable source code repository
• On-line forms for community input and catalogued responses
• Milestone schedule, project archives and history, related projects
• Management plan, board and team contacts
• More …
8 April 2003