Transcript Earth System Modeling Infrastructure Cecelia DeLuca/ESMF-NCAR March 31-April 1, 2009 CHyMP Meeting www.esmf.ucar.edu Outline • • • Elements of interoperability platforms Integrating across elements Summary www.esmf.ucar.edu.

Earth System
Modeling
Infrastructure
Cecelia DeLuca/ESMF-NCAR
March 31-April 1, 2009
CHyMP Meeting
www.esmf.ucar.edu
Outline
•
•
•
Elements of interoperability platforms
Integrating across elements
Summary
www.esmf.ucar.edu
Elements of interoperability platforms
1.
2.
3.
4.
5.
Tight coupling tools and interfaces
- hierarchical and peer component relationships
- frequent, high volume transfers on high performance computers
Loose coupling tools and interfaces
- generally peer-peer component relationships
- lower volume and infrequent transfers on desktop and distributed systems
Science gateways
- browse, search, and distribution of model components,
models, and datasets
- visualization and analysis services
- workspaces and management tools for collaboration
Metadata conventions and ontologies
- ideally, with automated production of metadata from models
Governance
- coordinated and controlled evolution of systems
www.esmf.ucar.edu
Tight coupling tools and interfaces
Examples:
• Earth System Modeling Framework (ESMF) - NASA, NOAA,
Department of Defense, community weather and climate models,
U.S. operational numerical weather prediction centers (HPC focus)
http://www.esmf.ucar.edu
• Flexible Modeling System (FMS) – NOAA precursor to ESMF, still
used at the Geophysical Fluid Dynamics Laboratory for climate
modeling
http://www.gfdl.noaa.gov/fms/
• Space Weather Modeling Framework (SWMF) – NASA-funded,
used at the University of Michigan for space weather prediction
www.esmf.ucar.edu
How coupling tools work:
• Users wrap their native data in framework data structures
• Users adopt standard calling interfaces for a set of methods that
enable data exchange between components
• Development toolkits help users with routine functions (regridding,
time management, etc.)
www.esmf.ucar.edu
ESMF: Standard interfaces
• Three ESMF component
methods: Initialize, Run, and
Finalize (I/R/F)
• Each can have multiple phases
• Users register their native I/R/F
methods with an ESMF
Component
• Small set of arguments:
AppDriver (“Main”)
Call Initialize
Call Run
Initialize
Call Finalize
Run
Finalize
Parent GridComp “Hurricane Model”
Call Initialize
Call Run
Initialize
Call Finalize
Run
Finalize
Child GridComp “Atmosphere”
Initialize
Run
Finalize
Child GridComp “Ocean”
Initialize
Run
Finalize
Child CplComp “Atm-Ocean Coupler”
call ESMF_GridCompRun (myComp, importState, exportState,
clock, phase, blockingFlag, rc)
www.esmf.ucar.edu
ESMF: Distributed data representation
1. Representation in index space (Arrays)
•
•
•
Simple, flexible multi-dimensional
array structure
Regridding via sparse matrix multiply with
user-supplied interpolation weights
Scalable to 10K+ processors - no global
information held locally
Supported Array distributions
2. Representation in physical space (Fields)
• Built on Arrays + some form of Grid
• Grids are: logically rectangular, unstructured mesh, or observational data streams
• Regridding via parallel on-line interpolation weight generation, bilinear or higher order
options
• Intrinsically holds significant amounts of metadata dynamic, usable for multiple purposes, limited annotation required
www.esmf.ucar.edu
ESMF: Coupling options
•
•
•
Generally single executable for simpler deployment
Push mode of data communication is very efficient
Coupling communications can be set up and called in a coupler, or called directly
from within components (for I/O, data assimilation)
•
Hierarchical components
for organization into subprocesses
Recursive components
for nesting higher
resolution regions
Coupling across C/C++
and Fortran
Ensemble management
•
•
•
www.esmf.ucar.edu
ESMF-based hierarchical
structure of GEOS-5
atmospheric GCM
ESMF: Performance portability
•
•
•
•
ESMF is highly performance portable, low (<5%) overhead
3000+ regression tests run on 30+ platform/compiler combinations nightly
See http://www.esmf.ucar.edu/download/platforms
Newer ports include native Windows, Solaris
Using TeraGrid Build and Test Service to simplify regression testing
Performance at the
petascale…
ASMM Run Time Comparison on XT4 (w/o -N1)
ASMM Run-Time Comparison
1000
3D Array
3D Array (-N1)
100
Bundle (-N1)
msec
10
0
Number of Processors
www.esmf.ucar.edu
16
38
4
81
92
40
96
20
48
10
24
51
2
25
6
12
8
64
32
16
8
1
4
Plot from Peggy Li, NASA/JPL
Tested on ORNL XT4,
-N1 means 1 core per node.
Bundle
Time (msecs)
Scaling of the ESMF sparse
matrix multiply, used in
regridding transformations, out
to 16K processors.
(ESMF v3.1.0rp2)
ESMF: Higher order interpolation
techniques in CCSM
Interpolation noise in the derivative of the zonal wind stress
Interp.
noise
grid index in latitudinal direction
Black = bilinear
Red = higher-order
ESMF v3.1.1
Green = higher order
ESMF v4.0.0
www.esmf.ucar.edu
• ESMF higher order interpolation
weights were used to map from a
2-degree Community Atmospheric
Model (CAM) grid to a POP ocean
grid (384x320, irregularly spaced)
• 33% reduction in noise globally in
quantity critical for ocean
circulation compared to previous
bilinear interpolation approach
• ESMF weights are now the CCSM
default
HAF
ESMF: Model map
SWMF
Legend
GAIM
Ovals show ESMF components and models
that are at the working prototype level or
beyond.
CCSM4
Dead atm
GFS
Data atm
GFS Atm Phys
POP Ocean
Dead ocean
Data ocean
Stub ocean
CICE ice
Dead ice
Data ice
Stub ice
CLM
Dead land
Data land
Stub land
GFS Atm Dynamics
NOAA
Department of Defense
University
NASA
Department of Energy
National Science Foundation
GFS I/O
FIM
NEMS
NMM-B Atm Dynamics
NMM-B Atm Phys
Ice sheet
ESMF coupling complete
ESMF coupling in progress
NMM History
Strat Chem
Component (thin lines)
Model (thick lines)
GEOS-5
Param Chem
GEOS-5 Atm Dynamics
GOCART
GEOS-5 GWD
GEOS-5 FV Dycore
FV Cub Sph Dycore
Tracer Advection
Land Info System
GSI
GEOS-5 Atm Physics
GEOS-5 Hiistory
GEOS-5 Atm Chem
GEOS-5 Radiation
GEOS-5 Aeros Chem
GEOS-5 LW Rad
GEOS-5 Solar Rad
GEOS-5 Surface
GEOS-5 Land
GEOS-5 Topology
GEOS-5 Veg Dyn
GEOS-5 Lake
GEOS-5 Catchment
GEOS-5 Moist Proc
GEOS-5 Turbulence
GEOS-5 Land Ice
GEOS-5 OGCM
Poseidon
GEOS-5 Data Ocean
GEOS-5 Salt Water
GEOS-5 Ocean Biogeo
WRF
POP
ROMS
MOM4
UCLA AGCM
COAMPS
MITgcm
MITgcm Atm
MITgcm Ocean
pWASH123
NCOM
www.esmf.ucar.edu
SWAN
CICE
HYCOM
ADCIRC
Loose coupling tools and interfaces
Examples:
• OpenMI
http://www.openmi.org
• Web service approaches
Coupling options:
• Generally multiple executable
• Pull mode of data communication simple but not efficient
(ask for a data point based on coordinates)
• Generally peer-peer component relationships
• Coupling across multiple computer languages (Python, Java, C++,
etc.)
www.esmf.ucar.edu
Science gateways – access centers
Examples:
• Earth System Grid (ESG) – DOE, NCAR, NOAA support, used to
distribute Intergovernmental Panel on Climate Change data and for
climate research
http://www.earthsystemgrid.org
• Hydrologic Information System (HIS) - NSF funded, used to
enhance access to data for hydrologic analysis
http://his.cuahsi.org
• Object Modeling System (OMS) - USDA effort, used for agricultural
modeling and analysis
http://javaforge.com/project/1781
www.esmf.ucar.edu
Metadata conventions and ontologies
Examples:
• Climate and Forecast (CF) conventions - spatial and temporal
properties of fields used in weather and climate
http://cf-pcmdi.llnl.gov
• METAFOR Common Information Model (CIM) – large EU-funded
project, climate model component structure and properties
(including technical and scientific properties)
http://metaforclimate.eu
• WaterML – Schema for hydrologic data developed by the
Consortium of Universities for the Advancement of Hydrologic
Science (CUAHSI)
http://his.cuahsi.org/wofws.html
www.esmf.ucar.edu
Governance
Pervasive issue in community modeling
Divergent effects of
• Multiple institutions
• Geographic dispersion
• Multiple domains of interest (working groups)
Must be balanced by strong integration body - strategies:
• Meets frequently enough to affect routine development (quarterly)
• Meets virtually to get sufficient representation
• Includes user and other stakeholder representatives
• Authorized to prioritize and set development schedule
• Supported by web-based management tools
www.esmf.ucar.edu
Integrating across interoperability elements
Examples from the Curator project (NSF and NASA)
• Automated output of CF and CIM XML schema from ESMF
(tight coupling + ontology)
• Ingest of ESMF-generated schema into ESG, propagation
into tools for search, browse, inter-comparison and
distribution of model components and models
(tight coupling + ontology + science gateway)
• Implementation of dataset “trackback” in ESG that connects
datasets with detailed information about the models used to
create the data
(tight coupling + ontology + science gateway)
• Implementation of personal and group workspaces in ESG
(science gateway + governance)
www.esmf.ucar.edu
Integrating across interoperability elements
(cont.)
• Translation of ESMF interfaces into web services to enable
invocation of ESMF applications from a science gateway, and enable
data and metadata from the run to be stored back to the gateway
(tight coupling + loose coupling + science gateway + ontology, new
TeraGrid funding)
Web service
interface
ESMF
interface
Tightly
coupled HPC
components
www.esmf.ucar.edu
Loosely
coupled
components
Issue of switch
from push to pull
data interactions…
Screenshot: Component trackback
www.esmf.ucar.edu
Screenshot: Faceted search
www.esmf.ucar.edu
Summary
• Cross-domain interoperability platforms have multiple elements
• Many of these elements already exist
• Integration activities (such as Earth System Curator) are the next
focus
www.esmf.ucar.edu
Image courtesy of Rocky Dunlap, Georgia Institute of technology