Transcript Components, Coupling and Concurrency in the Earth System Modeling Framework N.

Components, Coupling and Concurrency in the Earth System Modeling Framework
N. Collins/NCAR, C. DeLuca/NCAR, V. Balaji/GFDL, G. Theurich/SGI, A. da Silva/GSFC, M. Suarez/GSFC
What is ESMF?
•
•
ESMF provides tools for turning model codes into components with
standard interfaces and standard drivers, for use in application ensembles
and hierarchies
ESMF provides data structures and common utilities that components use
for routine services such as data communications, regridding, time
management and message logging
ESMF Superstructure
AppDriver
Component Classes: GridComp, CplComp, State
User Code
Infrastructure Timeline: ESMF Path
ESMF common modeling infrastructure (~5 year)
Technical foundation that allows for organized and exchangeable codes
Set of modeling codes that are based on ESMF (~5 year)
Community pool of interoperable science components with which to assemble applications
Multi-agency and international organization (~5 year)
Organizational foundation on which to establish collaborations and set priorities
Standards for model and data description (~10 year)
Prerequisite for an advanced modeling and collaboration environment that includes knowledge management
ESMF part of an established end-to-end community-based modeling environment (~20 year)
An Earth System Modeling Environment (ESME) that combines models, data, experiments, collaborative tools and
information resources in a way that fosters knowledge sharing and accelerates scientific workflow
Key Concepts
Registration
Data Classes: Bundle, Field, Grid, Array
Utility Classes: Clock, LogErr, DELayout, Machine
ESMF Community
ESMF Impacts
ESMF impacts a very broad set of research and operational areas that require
high performance, multi-component modeling and data assimilation systems,
including:
• Climate prediction
• Weather forecasting
• Seasonal prediction
• Basic Earth and planetary system research at various time and spatial scales
• Emergency response
• Ecosystem modeling
• Battlespace simulation and integrated Earth/space forecasting
• Space weather (through coordination with related space weather frameworks)
• Other HPC domains, through migration of non-domain specific capabilities
from ESMF –
facilitated by ESMF interoperability with generic frameworks such as CCA
Open Source Development
Open source license (GPL)
Open source environment (SourceForge)
Open repositories: web-browsable CVS repositories accessible from the ESMF website
http://www.esmf.ucar.edu
for source code
for contributions (currently porting contributions and performance testing)
Open development priorities and schedule: priorities set by users (previously during
meetings and telecons, transition to new Change Review Board), web-browsable
task lists
Open communication: frequent telecons and mailing list discussions
Open testing: all tests are bundled with the ESMF distribution and can be run by users
Open port status: results of nightly tests on many platforms are web-browsable
ESMF Current Challenges
Process and techniques for software requirements analysis, design and interface review with a
VERY large distributed community (installation of commercial software (DOORS) for
requirements archival and tracking, ties to testing and tasking)
Transition to multi-agency organizational structure
Design strategy for advanced grids and regridding; science and math hires to implement new
grids and regridding methods
Continued improvement of development and test processes
Clear, complete, carefully edited documentation and training program materials
New collaborations for technical and domain extension
Computational Characteristics of
Weather and Climate
Mix of global transforms and local communications
Load balancing for diurnal cycle, event (e.g. storm) tracking
Applications typically require 10s of GFLOPS,
100s of PEs – but can go to 10s of TFLOPS, 1000s of PEs
Required Unix/Linux platforms span laptop to
Earth Simulator
Platforms
Multi-component applications: component
hierarchies, ensembles, and exchanges
Data and grid transformations between
components
Applications may be MPMD/SPMD,
concurrent/sequential, combinations
Parallelization via MPI, OpenMP, shmem, combinations
Large applications (typically 100,000+ lines of source code)
•
•
•
•
•
Each box is an ESMF component
Every component has standard methods and interfaces to facilitate exchanges
New components can be added to the application systematically
Data in and out of components are state types that contain fields, bundles of fields
Coupling tools include regridding and redistribution methods
ESMF Interoperability Experiments
Coupling existing models for the first time
3 experiments done
• CCSM CAM with NCEP analysis
• GFDL FMS B-grid atmosphere with MITgcm ocean
• CCSM CAM with MITgcm ocean
5 additional experiments underway, with
• GFDL MOM4 ocean, LANL CICE, NCAR/NCEP WRF, NASA GMAO ocean and
atmosphere
PETs
1
2
3
4
5
6
7
8
9
AppDriver (“Main”)
Call Run
Run
GridComp “Hurricane Model”
LOOP Call Run
Run
GridComp
“Atmosphere”
Higher level Comp
Run
ESMF Framework Services
GridComp
“Ocean”
Run
CplComp
“Atm-Ocean Coupler”
cmp_register()
cmp_run()
cmp_init()
cmp_final()
Public subroutine
Private subroutine
Concurrent Execution
PETs
Standard Methods:
Initialize, Run, Finalize
1
2
3
Time
NSF National Center for Atmospheric Research
NOAA Geophysical Fluid Dynamics Laboratory
NOAA National Centers for Environmental Prediction
NASA Goddard Global Modeling and Assimilation Office
NASA Goddard Institute for Space Studies
NASA Jet Propulsion Laboratory
NASA Goddard Land Information Systems project
DOD Naval Research Laboratory
DOD Air Force Weather Agency
DOD Army Engineer Research and Development Center
DOE Los Alamos National Laboratory
DOE Argonne National Laboratory
University of Michigan
Princeton University
Massachusetts Institute of Technology
UCLA
Center for Ocean-Land-Atmosphere Studies
Programme for Integrated Earth System Modeling (PRISM)
Common Component Architecture (CCA)
Application Example: GEOS-5 AGCM
Components must provide a single externally visible entry point which will register
the other entry points with the Framework. Components can:
- Register one or more Initialization, Run, Finalize, and other standard entry
points.
- Register a private data block which can contain all data associated with this
instantiation of the Component; particularly useful when running ensembles.
Time
ESMF Infrastructure
Serial Execution
4
5
6
7
8
9
AppDriver (“Main”)
Call Run
Run
1
GFDL B-grid atmosphere
coupled to MITgcm ocean
Atmosphere, ocean, and
coupler are set up as ESMF
components
Uses ESMF regridding tools
3
2
Temperature SSI
import
Temperature SSI
export
Temperature
difference
GridComp “Hurricane Model”
AppDriver (“Main”)
LOOP Call Run
Call Initialize
NCAR Community
Atmospheric Model (CAM)
coupled to MITgcm ocean
Call Run
Call Finalize
Run
Initialize
Run
Run
GridComp
“Atmosphere”
Finalize
GridComp
“Ocean”
Parent GridComp “Hurricane Model”
Call Initialize
Atmosphere, ocean, and
coupler are set up as ESMF
components
Call Run
Run
Call Finalize
CplComp
“Atm-Ocean Coupler”
Initialize
Run
Finalize
Child GridComp “Atmosphere”
Uses ESMF regridding tools
Initialize
Run
Finalize
Child GridComp “Ocean”
Initialize
Run
Finalize
Child CplComp “Atm-Ocean Coupler”
Coupling Strategies
Hierarchies and
Ensembles
Pairwise
Hub and spokes
ESMF Component Overhead
Atmosphere
D
A
T
A
Land
DATA
AtmLandCoupler
DATA
Atmosphere
Land
DATA
Coupler
DATA
Ocean
Conclusions
D
A
T
A
ESMF encourages applications to be assembled
hierarchically
Coupling interfaces are standard at each layer
Components can be used in different contexts
sea ice
assim_atm
assim_atm
assim
atmland
atm
coupler
physics
ESMF supports ensembles with multiple
instances of components running
sequentially (and soon, concurrently)
Ensemble Forecast
Seasonal Forecast
ocean
SeaIce
land
dycore
assim_atm
assim_atm
• Measures overhead
of ESMF
superstructure in
NCEP Spectral
Statistical Analysis
(SSI), ~1% overall
• Run on NCAR IBM
• Runs done by JPL
staff, confirmed by
NCEP developers
•
•
•
•
•
Component approach powerful for integration of very large applications
Allows concurrent development in different domains and at different institutions
Allows common bodies of code to be used in different contexts with no change
Collective and ongoing design, review, and testing
Encourages concentration on science and not computational issues