Transcript WRF and WPS Compilation - DTC | Home | Developmental
WRF-DA Tutorial, 20-22 July, 2009, Boulder, CO
GSI Data Assimilation System
Hui Shao, Ming Hu, Laurie Carson, Louisa Nance, Xiang-Yu Huang, Bill Kuo
Developmental Testbed Center
John Derber, Russ Treadon
NOAA/NCEP/Environmental Modeling Center Acknowledgement: NOAA, AFWA, NCAR
Primary Developers
• NOAA/NCEP/EMC – John Derber, Jim Purser, Russ Treadon, Wan-Shu Wu, Dave Parrish, Lidia Cucurull, Dave Parrish, Manuel Pondeca, Paul van Delst, Daryl Kleist, Xiujuan Su, Yanqiu Zhu, and others • NASA/GMAO – Ricardo Todling, Ron Errico, Runhua Yang, Ron Gelaro, Wei Gu, and others • And NOAA/GSD, NCAR/MMM,… 2
Session Outline
• GSI System and Community Support (30 minutes, presented by Hui Shao) • Installation, Running, and Diagnostics (30 minutes, presented by Ming Hu) 3
GSI System
4
Gridpoint Statistical Interpolation (GSI)
•3D variational assimilation (same as WRF-VAR)
J =1/2 (x-x b ) T B -1 (x-x b ) + 1/2(H(x)-y 0 ) T (E+F) -1 (H(x)-y 0 ) + J C Fit to background + Fit to observations + constraints x x b B H y 0 E+F J C = Analysis (Output) = Background (Input) = Background error covariance (Input) = Forward model (within GSI) = Observations (Input) = R = Instrument error + Representativeness error (Input) = Constraint terms (within GSI)
5
Operational GSI Applications
System Physical SST retrieval NAM (regional) RTMA GFS (Global) HWRF Rapid Refresh (RR) AFWA operational Implementation date 9/27/2005 6/20/2006 8/22/2006 5/1/2007 6/19/2007 2010 2010 or 2011 * Collaborated with NASA/GMAO and others Primary Developers NCEP/EMC* NCEP/EMC* NCEP/EMC* NCEP/EMC* NCEP/EMC* NOAA/GSD NCAR/MMM 6
NAM HWRF Rapid Refresh Domain Air Force Weather Agency Domains 7
User input & initializations
GSI Code Flow
Read in & distribute observations, background & background errors Additional initializations Outer loop a) Set up right hand side of analysis equation (Compute observation innovations) b) Call inner loop i.
Compute gradient information ii.
iii.
iv.
v.
Apply background error Compute search direction Compute step size Update analysis increment 8 Write analysis & related output
Background Fields (x
b
)
• Currently works for the following systems
‒ Global • GFS • GMAO global ‒ Regional • WRF (ARW & NMM) - binary and netcdf • NCEP RTMA 9
Observations(y
o
)
• • • • • • • • • • • Radiosondes Pibal winds Synthetic tropical cyclone winds Wind profilers Conventional aircraft reports ASDAR aircraft reports MDCARS aircraft reports** Dropsondes MODIS IR and water vapor winds GMS, METEOSAT and GOES cloud drift IR and visible winds GOES water vapor cloud top winds • • • • • • • • • • • Surface land observations* Surface ship and buoy observation SSM/I wind speeds QuikScat wind speed and direction SSM/I precipitable water SSM/I and TRMM TMI precipitation estimates Doppler radial velocities VAD (NEXRAD) winds GPS precipitable water estimates GPS Radio occultation refractivity profiles SBUV ozone profiles (other ozone data under test) *Some of the data are restricted 10
•
Observations(y
o
) (cont.)
Regional – GOES-11 and 12 Sounders – thinned to 120km • Channels 1-15 Individual fields of view 4 Detectors treated separately Over ocean only – AMSU-A – thinned to 60km NOAA-15 Channels 1-10, 12, 15 NOAA-18 Channels 1-8, 10-11, 15 – AMSU-B/MHS – thinned to 60km NOAA-15 Channels 1-3, 5 NOAA-16 NOAA-17 Channels 1-5 Channels 1-5 NOAA-18 – HIRS – thinned to 120km NOAA-17 Channels 1-5 Channels 2-15 Global – GOES-11 and 12 Sounders –thinned to 180km Channels 1-15 Individual fields of view 4 Detectors treated separately Over ocean only – AMSU-A – thinned to 145km NOAA-15 Channels 1-10, 12-13, 15 NOAA-18 Channels 1-8, 10-13, 15 METOP Channels 1-13, 15 AQUA – AMSU-B/MHS – thinned to 240km NOAA-15 Channels 1-6, 8-13, 15 Channels 1-3, 5 NOAA-16 NOAA-17 Channels 1-5 Channels 1-5 NOAA-18 Channels 1-5 METOP – HIRS - thinned to 180km NOAA-17 Channels 1-5 Channels 2-15 METOP – AIRS – thinned to 180km AQUA Channels 2-15 148 Channels 11
Observation Operator (H)
• To use observation, GSI simulates observation using analysis variables – observation operator (
H
) Can be simple interpolation to ob location/time.
Can be more complex (e.g., radiation transfer).
For radiances, GSI uses CRTM.
12
Background Errors (B)
• Space correlation computed using recursive filters in horizontal and vertical • Multivariate relation • Flow dependent variability in background error – Background error variances modified based on 9 and 3 hour forecast differences: • Variance increased in regions of rapid change • Variance decreased in “calm” regions • Global mean variance ~ preserved – Being used for regional (US) surface analysis operationally.
13
HPC Surface Analysis
L b
) with flow dependent re-scaling rescaled
a
) without re-scaling “as is” Surface pressure background error standard deviation fields Valid at 00Z November 06, 2007 14
Option 2:
multivariate • Temperature (blue) increment forces increment in q (red).
• much smaller RH (shaded) increment.
Moisture analysis Option 1:
univariate • Temperature (blue) increment forces large increment in RH (shaded).
Observation Errors
• Improved specification of observational errors – Adaptive Tuning After tuning After tuning and smoothing Before tuning 16
Observation Quality Control
• External platform specific QC • Some gross checking in PREPBUFR file creation • Optimal interpolation quality control (OIQC) – on its way out • Analysis QC – Gross checks – specified in input data files – Variational quality control (VarQC) – implemented operationally in Feb 2009 Number of data rejected by OIQC VarQC weight (W) 17
Bias Correction of Radiance Data
• NCEP uses a two step process for T b – Scan angle correction – based on position – Air mass correction – based on predictors • Predictors – mean – path length (local zenith angle determined) – integrated lapse rate – integrated lapse rate ** 2 – cloud liquid water 18
B-O Histogram
19V 22V 37H 85H
DMSP15 July2004 : 1month before bias correction after bias correction 19
85V 19H 37V
Community Support
20
Community GSI Code
System Physical SST retrieval NAM (regional) RTMA Global HWRF RR AFWA operational Implementation date 9/27/2005 6/20/2006 8/22/2006 5/1/2007 6/19/2007 Early 2010 2010 or early 2011 Mode CRTM + analytical solution 3D-VAR 2D-VAR 3D-VAR 3D-VAR
DTC
Research Community
Goals:
•Provide current operational GSI capability to the research community (O2R) •Provide a framework for distributed development of new capabilities & advances in data assimilation (R2O) 21
Definition of Community Codes
• Free and shared resource • Ongoing distributed development by both research and operational communities • Maintained under version control • Periodic releases made available to the community • Includes latest developments of new capabilities and techniques • Centralized support • Provided in collaboration with developers
22
GSI Community Release Timetable
Tasks Beta release (Q1FY09) First release (Q1FY09) Tutorial Timeline Jun, 2009 Sep, 2009 Jun, 2010 Note Friendly user only (through the GSI website) With user support Residential tutorial and hand-on practical session 23
Community GSI Code Repository
NCEP EMC Repository Proposed Structure DTC
Code Management Plan
Developers
Community Repository
release Community Success of this structure will depend on communication and collaboration among all GSI developers, users, and DTC.
24
User’s Website
http://www.dtcenter.org/com-GSI/users/ • Release announcement • System component descriptions • Documentation – User’s Guide – Presentations • Registration • Software downloads • Bug fix reports • User support information – [email protected]
• Tutorial information 25
GSI User’s Guide
User’s Guide will be updated to be consistent with each new release to the community.
26
On-line Tutorial (Beta Release)
27
On-line Documents (Beta Release)
28
GSI Web Brower
29
O2R
2O2R2O2R2O2R2O …
30
GSI: Compile
Requirements
• System required libraries – FORTRAN 90/95 compiler – C compiler – Perl – netCDF • GSI system – Download GSI system tar files (
GSIbeta.tar.gz )
from –
http://www.dtcenter.org/com-GSI/users/index.php gunzip
and
untar tar –zxvf GSIbeta.tar.gz
(Should see
GSI/.
directory ) – cd to GSI directory
cd GSI
GSI Directory
Compile rules Compile scripts Source code directory Run directory
makefile README arch/ clean compile configure sorc fix/ lib/ include/ run/ makefile mains Top-level makefile General information about GSI code Directory where compile options are gathered script to clean created files and executables script for compiling WRF code script to configure the configure.wrf file for compile Directory for GSI and library source code Directory containing static/fixed data files Directory created for libraries Directory created for include/module files Directory for executable, and example script sorc bacio crtm_gfsgsi bufr sfcio libs gfsio sigio w3 mpeu makefile sp
33
Set environment
•
WRF_DIR
WRF needs to be compiled prior to compiling GSI
GSI uses WRF I/O API libraries to do file input and output
WRF directory specified:
setenv WRF_DIR /home/user/WRFV3
•
netCDF
If netCDF libraries are not located in the standard
/usr/local
, then
setenv NETCDF “path for netCDF”
For LINUX systems, make sure the
netCDF
libraries are installed using the same compiler (PGI, Intel, g95) that will be used to compile GSI.
Configuring GSI
• To create a GSI configuration file for your computer:
./configure
• This script checks the system hardware and software (mostly
netCDF
), and then offers the user choices for configuring GSI:
Choices for 32-bit LINUX operated machines are:
1. Linux i486 i586 i686, PGI compiler 2. Linux i486 i586 i686, Intel compiler 3. Linux i486 i586 i686, gfortran compiler
Choices for IBM machines are:
1. AIX xlf compiler with xlc
Configuring GSI, cont.
•
configure.gsi
– Created by the
./configure
command – contains compilation options, rules, etc.
– specific to your computer – can be edited to change compile options, if desired.
• At this time, the IBM option is well tested. Working on Linux option test. • The
arch/configure.defaults
file can be edited to add a new option if needed.
Compiling GSI • To compile:
./compile >& compile_gsi.log
• To get compile help message:
./compile -h
• If the compilation is successful, it will create one executable under bin
/
:
gsi.exe
Clean Compilation • To remove all object files and executables
./
clean
• To remove all built files, including
configure.gsi
./clean –a
Clean is recommended if compilation failed want to change configuration file
Running GSI
User’s Guide: Chapter 3 39
To run GSI, you need:
• GSI Executable
• Background (first guess) file • Observations
– Not needed for single observation experiment
• Fixed files (within GSI package) • Run script (namelist included)
40
Background
• GSI can use – WRF NMM input file in binary format – WRF NMM input file in netcdf format – WRF ARW input file in binary format – WRF ARW input file in netcdf format – GFS input file in binary format – GMAO global model input file in binary format • DTC has only tested regional analysis with WRF input – On IBM: both binary and nedcdf format – On Linux: netcdf format only 41
Observation
• All observations have to be in BUFR format • prepbufr: NCEP flavor BUFR – Need NCEP BUFR library 42
Fixed files • Collection of statistic and control files under fix directory
– Background and observation errors •
berror_stats, errtable
– Observation data control file (info files) •
convinfo , satinfo
– Bias correction used by radiance analysis •
satbias_angle, satbias_in
– Radiance coefficient used by CRTM •
EmisCoeff.bin, CloudCoeff.bin
43
Run script - structure
– Ask for computer resources to run GSI – Set environment variables for the machine – Set experiments variables (experiment name, analysis time, background and observation) – Check the definition of required variables – Generate a run directory for GSI (working or temp directory) – Copy GSI executable to run directory – Copy background file to run directory – Copy or link observations to run directory – Copy fixed files to run directory – Generate namelist for GSI – Run the GSI executable – Save the GSI analysis results 44
Run Script -
Experiment variables # analysis time (YYYYMMDDHH) ANAL_TIME =2008051112 # working direcotry, where GSI runs WORK_ROOT =./gsi/case # path and name of background file BK_FILE =./2008051112/bkARW/wrfout_d01_2008-05-11_12:00:00 # path of observations OBS_ROOT =./gsi/case PREPBUFR =./tutorialcases/data/newgblav.gdas1.t12z.prepbufr.nr
# path of fix files FIX_ROOT =./GSI/fix # path and name of the gsi executable GSI_EXE =./GSI/bin/gsi.exe
# which background error covariance and parameter will be used (GLOBAL or NAM) bkcv_option =NAM 45
Run Script - run directory
• Run directory – Create a new directory for each run • Copy or link data – Executable – Background • Must be copied, will be over-written by analysis result – Observations • Link or touch if not exist – Fixed files • Link 46
Run Script : namelist
• Generated by running scripts – Change namelist by editing run script • Works for both global and regional analysis • Detailed explanation in section 3.4
&SETUP miter =2, niter (1)=5, niter (2)=5, write_diag (1)=.true.,write_diag(2)=.false.,write_diag(3)=.true., qoption=1, gencode=78,factqmin=0.005,factqmax=0.005,deltim=$DELTIM, ndat=59,npred=5,iguess=-1, oneobtest =.false.,retrieval=.false.,l_foto=.false., use_pbl=.false., $SETUP / &GRIDOPTS JCAP=$JCAP,NLAT=$NLAT,NLON=$LONA,nsig=$LEVS,hybrid=.true., wrf_nmm_regional =.false., wrf_mass_regional =.true., diagnostic_reg=.false., filled_grid=.false.,half_grid=.true., netcdf =.true., regional =.true.,nlayers(63)=3,nlayers(64)=6, $GRIDOPTS 47
Tuning and Diagnostics
User’s Guide: Chapter 4 48
Standard out file - stdout • Includes lots of important information • First place to look after any GSI run
– If successful • Data distribution • Optimal iteration • Maximum and minimum of analysis fields – If fails, • which part of GSI has problem • what is the possible reason for failure 49
stdout - structure • read in all data and prepare analysis:
– read in configuration (namelist) – read in background – read in constant file (fixed file) – read in observation – partition background and observation data for parallel analysis
• optimal iteration (analysis) • save analysis result
50
stdout – example(1)
0: rmse_var=T 0: ordering=XYZ 0: WrfType,WRF_REAL= 104 104 0: ndim1= 3 0: staggering= N/A 0: start_index= 1 1 1 785191608 0: end_index= 69 64 45 -1603623772 0: k,max,min,mid T= 1 309.9411316
264.5114136
289.7205811
0: k,max,min,mid T= 2 310.6200562
269.5698547
295.0413208
……….
0: k,max,min,mid T= 45 510.0127563 472.5627441 494.7407227
Top of stdout: read in background End of stdout: write out analysis results 0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsua_n15.SpcCoeff.bin; 0: SpcCoeff RELEASE.VERSION: 7.01 N_CHANNELS=15 0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsub_n15.SpcCoeff.bin; 0: SpcCoeff RELEASE.VERSION: 7.01 N_CHANNELS=5 0: Read_SpcCoeff_Binary(INFORMATION) : FILE: hirs3_n16.SpcCoeff.bin; 0: SpcCoeff RELEASE.VERSION: 7.01 N_CHANNELS=19 0: Read_SpcCoeff_Binary(INFORMATION) : FILE: amsua_n16.SpcCoeff.bin; 0: SpcCoeff RELEASE.VERSION: 7.01 N_CHANNELS=15 Read CRTM coefficient 51
stdout – example (2)
6: READ_LIDAR: bufr file date is 2007 12 20 0 11: READ_AIRS: bufr file date is 2007 12 20 0 5: READ_RADAR: vad wind bufr file date is 2007 12 20 0 1: READ_PREPBUFR: bufr file date is 2007 12 20 0 9: READ_BUFRTOVS: bufr file date is 2007 12 20 0 mhsbufr 5: n,lat,lon,qm= 1 47.04 246.01 -9 -9 -9 -9 -9 -9 -9 -9 2 3 -9 3 -9 2 -9 4: READ_PREPBUFR: messages/reports = 872 / 97301 ntread = 1 Read observations 0: 0: GLBSOI: START pcgsoi 0: penalty,grad ,a,b= 1 0 0.160014047732364506E+05 0.724764165295960149E+06 0.132279098881E-02 0.00000000000E+00 0: pnorm,gnorm, step? 1 0 0.100000000000E+01 0.10000000000E+01 good 0: penalty,grad ,a,b= 1 1 0.150427003330115986E+05 0.608359222451359266E+06 0.124401271687E-02 0.83938151711E+00 0: pnorm,gnorm, step? 1 1 0.940086232815E+00 0.83938982252E+00 good Iteration 0: PROGRAM GSI_ANL HAS ENDED . IBM RS/6000 SP 52
Single Observation Test
• A good way to check – ratio of background and observation variance – pattern of background error covariance • Setup in namelist – SETUP oneobtest=.true., – SINGLEOB_TEST maginnov=1.0, magoberr=1.0, oneob_type='t', oblat=20., oblon=285., obpres=850., obdattim= 2007081500, obhourset=0., 53
Control Data Usage
• From namelist and the links of observation files dfile(01)='prepbufr‘, dtype(01)='ps‘, dplat(01)=' ', dsis(01)='ps' dfile(02)='prepbufr‘ dtype(02)='t', dplat(02)=' ', dsis(02)='t', dfile(27)='amsuabufr‘,dtype(27)='amsua‘,dplat(27)='n15‘,dsis(27)='amsua_n15', dfile(28)='amsuabufr‘,dtype(28)='amsua‘,dplat(28)='n16‘,dsis(28)='amsua_n16', • From info file ( convinfo otype …) type sub iuse twindow numgrp ngroup nmiter gross ermax ermin var_b var_pg ps ps ps ps ps ps ps ps 111 0 -1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000
132 0 -1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000 180 0 1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000
181 0 1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000
182 0 1 1.5 0 0 0 5.0 3.0 1.0 10.0 0.000
183 0 -1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000
187 0 1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000
188 0 -1 0.25 0 0 0 5.0 3.0 1.0 10.0 0.000
t 120 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000
t 126 0 -1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000
t 130 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000
t 131 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000
t 132 0 1 1.5 0 0 0 7.0 5.6 1.3 10.0 0.000
54
Observation fit - statistic files
55
Convergence Information
• stdout 0: 0: GLBSOI: START pcgsoi 0: penalty,grad ,a,b= 1 0 0.160014047732364506E+05 0.724764165295960149E+06 0.132279098881E-02 0.00000000000E+00 0: pnorm,gnorm, step? 1 0 0.100000000000E+01 0.10000000000E+01 good 0: penalty,grad ,a,b= 1 1 0.150427003330115986E+05 0.608359222451359266E+06 0.124401271687E-02 0.83938151711E+00 0: pnorm,gnorm, step? 1 1 0.940086232815E+00 0.83938982252E+00 good • More detail in fort.220
J = 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.111428969204742752E+05 0.253237990246203207E-01 0.225585966101596478E+04 0.262756287766784135E+04 0.256346518803503204E+04 0.921301367380113305E+02 0.244005730835839540E+04 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 0.000000000000000000E+00 20 items in cost function: 5 wind observation 6 satellite radiance observation 7 temperature observation 8 precipitable water obs. 9 specific humidity obs. 20 surface pressure observation 56
Questions, Comments, Suggestions 57