Transcript Using The GMAO GEOS-5 AGCM Software Integration and Visualization Office NASA/GSFC

Using The GMAO GEOS-5 AGCM
Software Integration and Visualization Office
NASA/GSFC
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
About GEOS-5
• Developed by the NASA/GSFC Global
Modeling and Assimilation Office
(http://gmao.gsfc.nasa.gov)
• First model to be constructed “ground up”
based upon Earth System Modeling
Framework
• Designed for weather & climate research
• Makes use of finite-volume dynamical core
(Lin, 2004)
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Outline
•
•
•
•
•
•
•
•
•
•
•
•
•
•
Model Overview
Getting the code
Directory Structure
Browsing the code (Understand for Fortran)
Setting your Environment
Building the code
Setting up an experiment
The run.script
Submitting/Monitoring Jobs at NCCS
Diagnostics
Post-Processing
Verification
Quick Start Guide
Where to get help
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Let’s Get an interactive job in
the Training Queue
• Interactive Batch
qsub -I -l ncpus=4 -l walltime=8:00:00 -W
group_list=k3002 -q training /usr/bin/tcsh
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
GEOS-5 Overview from the GMAO
•The Goddard Earth Observing System Model, Version 5 (GEOS-5) is a system of
models integrated using the Earth System Modeling Framework (ESMF).
•The system includes an atmospheric general circulation model (AGCM) developed
for both weather and climate applications using the finite-volume dynamical core
(Lin, 2004).
•Developments of GEOS-5 were guided by the need for realistic representation of
tracer transports and stratospheric dynamics and the need for a realistic
representation of moist processes for climate applications and for atmospheric data
assimilation.
•The GEOS-5 AGCM is coupled to the Catchment Land Surface Model (LSM).
•The GEOS-5 atmospheric data assimilation system (DAS) couples the AGCM with
the Grid-point Statistical Interpolation (GSI) analysis scheme being developed by
the Environmental Modeling Center of the National Centers for Environmental
Prediction (NCEP/EMC) and GMAO.
•The AGCM and the DAS have been run with 72 levels to 0.01 hPa and at horizontal
resolutions from 2° to 0.25 °.
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
GEOS-5 Overview from the GMAO
•The Aerochem component of GEOS-5 incorporates parameterized tropospheric
chemistry, the stratospheric chemistry and GOCART aerosol modules from the
Atmospheric Chemistry and Dynamics Branch at GSFC as well as a carbon
transport module.
•The GEOS-5 Coupled Climate General Circulation Model (CGCM) couples the
AGCM with the Poseidon V5 ocean model developed by Paul Schopf at George
Mason University.
•The GMAO Ocean Data Assimilation System (ODAS) is fully integrated into GEOS5 using ESMF couplers. Other chemistry and ocean models will be included over
time.
•The GEOS-5 architecture supports running each of the component models in
stand-alone mode as well as in various coupled configurations.
•The GEOS-5 system undergoes continual development to support the scientific
investigations of GMAO and collaborators.
•Major new releases can be expected every 6-12 months, with minor upgrade
releases for bug fixes as needed.
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
GEOS-5 GCM Documentation
• http://gmao.gsfc.nasa.gov
– http://gmao.gsfc.nasa.gov/systems/geos5/test.php
– http://gmao.gsfc.nasa.gov/systems/geos5/index_arch.php
• http://map.nasa.gov
– http://map.nasa.gov/GMAO_GEOS5.html
– http://map.nasa.gov/GEOSGCMbrowser/index.html
– http://map.nasa.gov/GEOSGCMbrowser/GEOSgcm_5_lvls.jpg
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
GEOS-5 GCM Structure
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Getting the Code
• Sourcemotel / CVS
– cd $HOME
– cvs -z3 -d:ext:[email protected]:/cvsroot/esma co -r
GEOSagcm-Eros-beta7p12 GEOSGCM
• Tarball from MAP website
– cd $HOME
– tar -zxvf GEOSagcm-Eros-beta7p12.tgz
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
src
src/GNUmakefile
GEOSGCM
Model Directory Structure
src/Config
src/Application
src/Shared/GEOS_Shared
src/Shared/GEOS_Base
src/Shared/GEOS_Util
src/Shared/GEOS_Util/post
src/Shared/GEOS_Util/plots
src/Shared/GMAO_pilgrim
src/GEOShistory_GridComp
src/GEOSgcm_GridComp
src/GEOSgcm_GridComp/GEOSogcm_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp/FVdycore_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSsuperdyn_GridComp/GEOSgwd_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSturbulence_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp/GEOSsolar_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSradiation_GridComp/GEOSirrad_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSmoist_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSchem_GridComp/GEOSpchem_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp
src/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Understand for Fortran
• What is it?
– An interactive development environment tool providing
reverse engineering, automatic documentation, metrics and
cross referencing of FORTRAN source code.
• Uses
–
–
–
–
–
Browsing source code
Understanding connections in the code
Understanding variable declaration/usage
Viewing use/use-by/invocation trees
Generating in-depth reports
• References
– http://www.scitools.com/uf.html
– http://scitools.com/documents/manuals/pdf/understand_f.pdf
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Creating your development branch
Make a branch:
You have checked out a “tagged” version. You cannot commit changes to a “tagged” version, since that
would make the tag meaningless. Before committing your changes, you must create a new “branch” and
then commit to the branch:
cvs tag –b Branch_Name $files
where $files is the list of files that you have modified, and Branch_Name is a name of your choice. The
branch name should be somewhat descriptive, including things such as the tag you are branching from
(eg., b_GEOSagcm-Eros-beta7p12) or the name of a control experiment (eg., b_e0247).
Once the branch has been created, you can “update” your files to that branch:
cvs upd -r Branch_Name $files
This does not change the files you modified, only their cvs “status” in your sandbox.
Commit:
To commit your changes, simply type:
cvs commit $files
Tag:
Finally you are ready to tag ALL files (newly modified as well as untouched) to reflect your new
experiment. At the GEOSGCM/src level:
cvs tag Your_Tag_Name
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Building on Palm/Explore at NCCS
• Set Module Environment (preferably within .cshrc or .login):
–
–
–
–
module purge
module load intel-comp.8.1.034
module load mpt.1.12.0.nas
module load scsl.1.5.1.1
–
–
–
setenv BASEDIR /share/ESMA/baselibs/v2_2rp2.34meta
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/share/ESMA/baselibs/v2_2rp2.34meta/Linux/lib
setenv GEOSUTIL ~/GEOSGCM/src/Shared/GEOS_Util
• Issue install command:
–
–
cd ~/GEOSGCM/src
gmake install
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
• Installation
Building the Code
– gmake install builds the executable
• Dependencies
– Generated during the build process
– May reduce recompilation times after changes
• Errors/Warnings
– Warnings
/home1/wputman/GEOSagcm-Eros-beta7p12/GEOSGCM/Config/GMAO_base.mk:160: WARNING: No environment check performed.
fortcom: Warning: mpi_isend.F90, line 12: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.
[REQUEST]
comm, request, ierror)
-------^
fortcom: Warning: mpi_recv.F90, line 11: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.
[STATUS]
subroutine mpi_recv( buf, count, type, pe, tag, comm, status, ierror)
----------------------------------------------------------^
fortcom: Warning: mpi_irecv.F90, line 11: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.
[REQUEST]
subroutine mpi_irecv( buf, count, type, pe, tag, comm, request, ierror)
-----------------------------------------------------------^
fortcom: Warning: cloudnew.F90, line 100: A dummy argument with an explicit INTENT(OUT) declaration is not given an explicit value.
[CLDNCCN]
CLDNCCN
, &
--------------------------^
• Where is my executable?
– ~/GEOSGCM/src/Application/GEOSgcm.x
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Setting up an Experiment
• Experiment Directory
mkdir -p $HOME/geos5/e0247
cd $HOME/geos5/e0247
cp $HOME/GEOSGCM/src/Application/run.script .
edit run.script (change group ID from g0620 to …[k3002])
• Run Directory (Work/Scratch space)
– Work directory
mkdir /nobackup/$USER/e0247
cd /nobackup/$USER/e0247
ln -s $HOME/GEOSGCM/src/Application/GEOSgcm.x .
cp /nobackup/wputman/GEOS-5.0_restarts/* .
– Scratch directory (where the model runs…)
/nobackup/$USER/e0247/scratch (ONLY if $TMPDIR is not defined on the system)
• Mass Storage
– Created by run.script
/output/$USER/GEOS5.0/e0247
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Getting to know the run.script
• Basic modifications
–
–
–
–
–
–
–
–
processor count
MPI layout
resolution
timestep
begin/end dates
Job segments (YYYYMMDD HHMMS)
Iterations/counters
diagnostics
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Running Your Experiment
• PBS
– Batch
Set appropriate group ID
qsub run.script
– Interactive
qsub -I -l ncpus=32 -l walltime=2:00:00 -W group_list=your_gid /usr/bin/tcsh
• Monitoring job progress
qstat -s
184932.palm
user_id general e0247
28266 -- 120 398540 06:00 R 01:21
Job run at Tue Nov 21 at 17:12 on e1:ssinodes=120:ncpus=120:mem=4798464...
ssh e1 “ls -l /pbs/spool/*”
ssh e1 “tail -f /pbs/spool/184932.palm.OU”
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Diagnostics
•
Where are they defined
– History.rc file created in run.script
– Example
list(1)%filename: 'geosgcm_prog',
list(1)%template: '%y4%m2%d2_%h2%n2z',
list(1)%format: 'CFIO',
list(1)%frequency: 060000,
list(1)%fields: 'PHIS'
, 'SUPERDYNAMICS',
'SLP'
, 'DYN'
,
'U'
, 'DYN'
,
'V'
, 'DYN'
,
'T'
, 'DYN'
,
'PS'
, 'DYN'
,
'DELP'
, 'DYN'
,
'OMEGA'
, 'DYN'
,
'QV'
, 'MOIST'
,
'H2O_PROD' , 'CHEMISTRY'
,
'RH2'
, 'MOIST'
, 'RH' ,
::
•
An experiment may have Multiple output streams
–
–
Varying formats (binary [‘raw’] or HDF [‘CFIO’])
Consistent vertical dimension
•
•
•
•
1 to LM for vertical centered variables
1 to LM+1 for edge variables
surface or 2D variables
The default output streams in run.script are required for the GEOS_Util postprocessing plotting utility to execute cleanly.
–
Latest Update:
August 14, 2007
Post-processing of eta to pressure level conversion should still occur
GEOS-5 AGCM Tutorial
•
Options for Output Streams
–
–
–
–
–
–
filename
–
Character string defining the filename of a particular diagnostic output stream.
–
Character string defining the time stamping template following GraDS conventions.
–
Character string defining the file format (‘flat’ of ‘CFIO’) ‘flat (raw binary)’ is default, ‘CFIO’ is HDF.
–
Character string equal to "instantaneous" or "time-averaged". Default = "instantaneous".
–
Character string equal to the list description. Defaults to "expdsc".
template
format
mode
descr
frequency
–
–
–
–
–
–
–
–
–
Latest Update:
August 14, 2007
Integer (HHMMSS) for the frequency of output. Default = 060000.
acc_interval
–
Integer (HHMMSS) for the acculation interval (<= frequency) for time-averaged diagnostics.
–
Integer (YYYYMMDD) reference date from which the frequency is based.
–
Integer (HHMMSS) reference time from which the frequency is based. Default = 0.
ref_date
ref_time
end_date
–
–
Diagnostics
Integer (YYYYMMDD) ending date to stop diagnostic output. Default is disabled.
end_time
–
Integer (HHMMSS) ending time to stop diagnostic output. Default is disabled.
–
Integer (HHMMSS) for the duration of each file. Default = 240000 (one-day segments).
–
Paired character strings for the diagnostic Name and its associated Gridded Component.
duration
fields
resolution
–
Optional resolution (IM JM) for the ouput stream. Default is the native resolution.
–
Optional list of output levels (1 to LM for vertical centered, 1 to LM+1 for edge vars). Default is all levels.
levels
GEOS-5 AGCM Tutorial
Run Directory Structure (after execution)
• /nobackup/$USER/$EXPID
– Scratch (if $TMPDIR does not exist)
– restarts
– listings
– Holding (daily eta and prs output streams)
– Output streams (monhtly averages):
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
drwxr-xr-x
Latest Update:
August 14, 2007
2 wputman k3002
2 wputman k3002
2 wputman k3002
2 wputman k3002
2 wputman k3002
2 wputman k3002
2 wputman k3002
2 wputman k3002
10 2006-11-20 18:46 geosgcm_edge
10 2006-11-20 18:46 geosgcm_gwd
10 2006-11-20 18:46 geosgcm_moist
10 2006-11-20 18:46 geosgcm_prog
10 2006-11-20 18:46 geosgcm_rad
10 2006-11-20 18:46 geosgcm_surf
10 2006-11-20 18:46 geosgcm_tend
10 2006-11-20 18:46 geosgcm_turb
GEOS-5 AGCM Tutorial
Output Directory Structure
• /output/$USER/GEOS5.0/$EXPID
– restarts
– output streams
• Including eta and pressure level files
– monthly
– Plots
• All output streams are GrADS readable
– HDF files use sdfopen …
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Post-Processing
• GEOS_Util
– eta2prs
• Based on History.rc file
– Monthly means
– Archiving
– Plotting
• Some visualization capability is provided in the
post-processing scripts
• Quickplot routine produces seasonal mean plots
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Quick Start Guide at NCCS
• Create Source Directory
cd $HOME
cvs -z3 -d:ext:[email protected]:/cvsroot/esma co -r GEOSagcm-Eros-beta7p12 GEOSGCM
• Set Module Environment (in .cshrc or .login)
module purge
module load intel-comp.8.1.034
module load mpt.1.12.0.nas
module load scsl.1.5.1.1
setenv BASEDIR /share/ESMA/baselibs/v2_2rp2.34meta
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}:/share/ESMA/baselibs/v2_2rp2.34meta/Linux/lib
setenv GEOSUTIL ~/GEOSGCM/src/Shared/GEOS_Util
• Build
cd GEOSGCM/src
gmake install
• Experiment Directory (scripts)
mkdir -p $HOME/geos5/e0247
cd $HOME/geos5/e0247
cp $HOME/GEOSGCM/src/Application/run.script .
edit run.script (change group ID from g0620 to [k3002], confirm resolution, date, execution intervals..)
• Run directory (work/scratch space)
mkdir /nobackup/$USER/e0247
cd /nobackup/$USER/e0247
ln -s $HOME/GEOSGCM/src/Application/GEOSgcm.x .
cp $RESTARTS .
• Submit the job
cd $HOME/geos5/e0247
qsub run.script
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Verification
• A 10-day sample case
– Typical STDOUT
– HDF eta or pressure level data
– GrADS readable
• A long integration
– Monthly means
– Plotting
• Horizontal contour plots
• Comparisons with NCEP/ECMWF Reanalyses
• Taylor Diagrams
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Sample plots
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Std Deviation
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
RMS Error on
Green Curves
Std Deviation
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Sample plots
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
QuickTime™ and a
TIFF (LZW) decompressor
are needed to see this picture.
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Visualization Support
• All GEOS-5 output streams are GrADS readable
• There are some built-in utilities for generating
monthly-mean plots in GEOS_Util using GrADS
• NCCS/NAS systems provide access to IDL for
visualization
• SIVO and the Scientific Visualization Studio will
provide support to collaborators with these tools
as well as via WMS-based tools
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial
Where to Get Help
• MAP website
– GEOS-5 Software Page
• http://map.nasa.gov/GMAO_GEOS5.html
– GEOS-5 FAQs
• To be developed on MAP website…
• GMAO website
– Experiments (GMAO internal access required)
• http://gmao.gsfc.nasa.gov/systems/geos5/index_results.php
– Technical Documentation
• http://gmao.gsfc.nasa.gov/systems/geos5/index_arch.php
– Scientific Documentation
• http://gmao.gsfc.nasa.gov/systems/geos5/index_pubs.php
Latest Update:
August 14, 2007
GEOS-5 AGCM Tutorial