Transcript Development of an Integrated Earth System Climate Modeling Environment Purdue University/RCAC NCAR/ESG NOAA/CIRES A Great Team! Purdue/RCAC Carol X.

Development of an Integrated Earth
System Climate Modeling Environment
Purdue University/RCAC
NCAR/ESG
NOAA/CIRES
A Great Team!
Purdue/RCAC
Carol X. Song, Lan Zhao, Chris Thompson, Robert Campbell,
Jungha Woo, Han Zhang, Madhavan Lakshminarayanan
NCAR
Don Middleton, Nathan Wilhelmi, Eric Nienhouse, Michael
Burek, Rick Brownrigg
NOAA/CIRES
Kathy Saint (SGI), Cecelia DeLuca, Sylvia Murphy
“Developing an integrated end-to-end TeraGrid climate
modeling environment”, TeraGrid 2011 conference, Salt Lake
City, UT, July 2011.
05/11/2012
XSEDE Science Gateway
2
Background
• Motivation for a climate modeling environment





CCSM/CESM
Resource
Expertise
A collaborative online environment
Existing synergistic community efforts
Earth System Grid, Earth System Curator, ESMF, METAFOR
Common Information Model and Purdue CCSM3 portal
• Project goals
 Provide an integrated environment for climate modeling,
data/metadata archival, visualization and analysis
 Support research and education, lower barriers to using
climate models
05/11/2012
XSEDE Science Gateway
3
System Architecture
CESM portal
ESG gateway
Account DB
User
requests
Token Mgr
TG MyProxy
CESM Web Services
Data/
Metadata
Create
Case
Configure Case
Authentication/
Authorization
Debugging
Transfer
Files
Post-process
Output
05/11/2012
ESG Data
Publisher
Submit Case
Scratch Storage
Track
Status
SOAP and
Restful
XML/JSON
Job
Management
Publish
Data
Output
iRODS
XSEDE Science Gateway
Publish
Metadata
Jobs
XD community
account/Individual
account
4
CESM Portal - Model Workflow
CESM Portal
• Insulate users from
cluster
SOAP
JBOSS
Job Scripts
Create case
Configure case
Compile case
Publish Scripts
CESM-WS/
gateway
CESM Model
Create
create_case
Configure
./configure
Compile
<case>.compile
Execute
<case>.submit
Query Status
Execute case
Data
Configure postprocess
Run post-process
Post-processing
Post-processing
Engines
AMWG
NCL scripts
ESG Publication
workflow
05/11/2012
…
XSEDE Science Gateway
 Hide command line
details
 Integrate CESM 1.0
+ metadata
capabilities + ESG
publishing + NCL
 Web services (SOAP,
Restful XML/JSON)
• Manage user’s
experience
 Maintain job status
and metrics
 End-to-end
modeling from
simulation to
data/metadata
publishing
 Bind independent
processes into
unified workflow
5
ESMF Attributes and Self-describing Models
• Metadata is increasingly important in Earth science modeling
 Understand multi-component models with many options and parameters
 Distinguish simulations from one another
 Archive simulations with full descriptions for preservation and
defensibility
 Enable run reproducibility
• ESMF Attributes store, aggregate, and output metadata
 Structured as name-value pairs
 Arranged in packages by purpose and convention:
International Standards Organization (ISO), Climate and Forecast (CF),
METAFOR Common
Information Model (CIM), etc.
 Output in XML and other formats
• ESMF Attributes in CESM
 CESM components (land, atm, ice, ocean, sea ice) and driver are being
instrumented with Attributes
 Information captured includes component level, field level, and
system level metadata
05/11/2012
XSEDE Science Gateway
6
CESM Metadata in the ESG Gateway
Implemented in the model
by ESMF and Curator.
Screenshot of CESM metadata
Exported by CESM at run
time.
Uploaded to Purdue’s atom
feed.
Ingested and displayed by
ESG.
05/11/2012
XSEDE Science Gateway
7
Data Discovery, Access and Analysis
• ESG: Science Gateway for Earth System Data Access








Data discovery and access across federation of data nodes.
Identity services and project specific access control.
Data publishing services.
Versioning and provenance support.
Bulk data download, wget scripts.
Data use tracking and metrics.
Model documentation metadata browsing.
ATOM CIM feed and ESG ingest.
• NCL Backend to Product Services




05/11/2012
Integrated 64-bit NCL visualization engine.
Big data capable analysis functions.
ParNCL as a drop-in replacement, for parallel computation.
Plug-in component to NOAA/PMEL LAS Product Server.
XSEDE Science Gateway
8
Demo
05/11/2012
XSEDE Science Gateway
9
Demo
05/11/2012
XSEDE Science Gateway
10
Demo
05/11/2012
XSEDE Science Gateway
11
Next Steps
• Transition into XD
 Release CESM portal
 Work with faculty and instructors to use in teaching
 Need help in connecting with users
• Federated authentication/identity between CESM and
ESG gateways
• Making a rich set of tools for analysis and visualization
available to CESM gateway users
• Add other resources to backend for better user
experience
05/11/2012
XSEDE Science Gateway
12