NARR User Issues

Wesley Ebisuzaki CPC/NCEP [email protected]

http:wwwt.emc.ncep.noaa.gov/mmb/rreanl/

Topics

● Files: contents, availability, formats ● Tools: BUFR, GRIB ● Obtaining the data: NCEP, NCAR, NCDC ● Scripting downloads ● What is reality? Is reanalysis truth? An insider's viewpoint.

GRIB Files

● For grids (horizontal fields) such as 500 mb Z ● Lambert conformal grid: 32 km eta grid (#221) – Have grids of lat/lon, display programs ● Model grid: not available except at NCEP ● – No plans to make available, display programs?

Analyses are on-line and forecasts ?

● 99% of people will use the GRIB files

BUFR

● For all input observations except SST, snow, ice, observed precipitation (GRIB) ● Input obs BUFR files have QC marks ● Analyses/forecasts at station locations (MOLTS) – Much smaller than GRIB files, limited locations ● Distribution (informal) – Includes restricted data – Few users (experience from global reanalysis) – Often other sources of data (in easier formats)

Misc Files

● Restart files: huge, full precision, NCEP-only ● Statistics: – Fit of obs to NARR, fit of obs to R2 ● – Small, can be made available on-line Observation counts: – Small, should be put on-line ● PNG (monitoring plots) – Small, should be put on-line

Want more files?

Don't expect too much.

NARR has zero funding.

Despair?

● Global Reanalysis has been running this way for years!

● CPC has been running global reanalysis for its own operations. CPC and the outside community have shared interest in maintaining a stable assimilation system. NCEP has an interest in others using its products.

● However, limited resources, need to be selective, work with partners

Status

● Real-time is running (4 days late) ● Started building 1.5-2 year on-line rotating archive of latest analyses at NCEP ● 1979-2003 merged files at NCAR, NCDC ● NCDC has 5 TB merged files on-line, trying to support the on-line community by NOMADS ● EMC people are working on other projects

Tools: BUFR

● Situation improving ● GrADS can read bufr directly

Tools: GRIB

● GrADS: display and computations – grib2ctl.pl: make GrADS control files ● wgrib: inventory, decode GRIB files, remove records from a file (save space) ● lcgrib: regional subset (save space) ● copygb: change grid (save space or neurons) ● grbpoint.pl: grib -> point value printout (slow) ● Gempak (?), NCL (?)

Example: GrADS

● Make control file (analyses) – grib2ctl.pl -rr cpc_rr_mon.197901 >my.ctl

– grib2ctl.pl -rr cpc_rr_mon.%y4%m2 >my.ctl

● Make index file (analyses) ● – gribmap -0 -i my.ctl

Run GrADS – grads – open my.ctl

Example: wgrib

● Use: inventories, make files smaller, decode data for fortran and C programs ● See contents of a grib file – wgrib (options) cpc_mon_197901 ● To extract records, decode records, see wgrib home page.

Example: lcgrib

● Regional subset of Lambert conformal grid ● *** MAKE SMALLER FILES *** ● lcgrib in-grib out-grib west south east north – West, south, east and north are in degrees – N and E are +ve, S and W are -ve

Obtaining Data: NCEP

● Rotating archive: latest 1.5 – 2 years of analyses ● – http://nomad7.ncep.noaa.gov/ncep_data Other files: TBD ● NCEP not set up to deal with the public requests – Go through the web – Go through NCDC, NCAR

Obtaining Data: NCAR

● Have 1979-2003 merged data ● Sure they will get more of the data ● Plans, availability?

Obtaining Data: NCDC

● NCDC has the merged data 1979-2003 and want 2004 ASAP ● NCDC will use NOMADS to distribute the data to the on-line community.

● NOMADS is project of many institutions including NCDC and NCEP for distributing data over the internet.

● NOMADS-NCDC is taking software from NCEP and COLA to distribute GRIB data.

Data Access at NCDC

● Custom plots ● ftp4u: access by selected time, field, region ● http access to raw data (like FTP) ● GDS (GrADS Data Server): OpeNDAP server (formerly DODS), on-line access to data

Difficulties in on-line distribution

● Downloading a 5 TB data set is difficult ● Need ways to minimize data transfer ● Plots (maps) work but limited utility ● GDS needs to be optimized for large datasets ● ftp4u is a solution

FTP4U

● Only download data that you need – Select levels – Select variables – Select geographic regions – Select times

Making FTP4U Work

● Don't request more data than needed ● Make one big request, not many small requests of the same file; i.e., make one request to get the winds, temp and heights rather than 3 requests. Do not request long time series.

● Shared resource (1 job at a time) ● Be prepared to script/automate the downloads – May be impossible to download in a few hours, may require days, weeks and multiple jobs/requests

Scripting FTP4U Downloads

● FTP4U is normally accessed by a browser ● Anything a browser can do, a script can also do ● Experience at NCEP is that most plot/ftp4u access is scripted (automated) ● NCDC's modifications to the NCEP provided programs make it more difficult to script the download but nothing that a cURL user couldn't handle. NCDC wants to change!

Scripting Downloads cont'd

● Use cURL, a unix/windows program, to “emulate” a browser ● Each browser page becomes a call to cURL.

● With a little RTFM was able get script working ● sample program will be put on NARR home page

Scripting Downloads cont'd

● Downside: steep learning curve, may have no local experts for help ● Upside: can make large data transfers possible, can be used in other projects (ex. getting fcst data automatically, skewing on line opinion polls) ● Given that we get more data off the web, it's a skill that someone in a large group should eventually have.

Obtaining Data: NCDC

● It's a learning situation for NCDC ● Give feedback (complain), it's the only way to improve the situation. Lessons learned here will help the next time you try getting a different dataset from NCDC's nomad server.

● Current issues: full disk (new RAID is will be installed soon), obtaining time series, scripting support

What is Reality?

● Some people consider reanalysis to be truth ● NARR is new, bound to be warts, already found some.

What is Reality?

● Accuracy of reanalysis depends on many factors – Data going into the analyses ● Types, counts, quality of observations – Accuracy of previous analysis – Accuracy of atmospheric/land model – Spatial and temporal filtering – NO EASY ANSWER

What is Reality? cont'd

● Accuracy – check to the appropriate data counts and biases ● Need to add our counts to web ● Need to add fit to data to web – check other reanalyses – Fit to observations – Consider sensitivity to modeling assumptions – Sensitivity experiments (data, resolution, model), only available to limited # of people

What is Reality? cont'd “Trust but verify”

Ronald Reagan

URLs

● copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb – change grid type of grib files ● copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb – change grid type of grib files ● cURL: http://curl.haxx.se

– download URLs, handles post data ● GEMPAK: http://my.unidata.ucar.edu/content/software/gempak/index.html

– graphics, can handle NARR files with some effort ● GrADS: http://grads.iges.org/grads/ – graphics, computations

URLs

● grbpoint.pl: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/wgrib.scripts

– simple program to extract data from lat/lon ● grib2ctl.pl: http://www.cpc.ncep.noaa.gov/products/wesley/grib2ctl.html

– makes control files for GrADS ● g95: http://www.g95.org/ – f95 compiler for copygb ● HIPHOP: http://iacweb.etzh.ch/stall/dominik/hihop/ – grib support for idl ● lcgrib: http://www.cpc.ncep.noaa.gov/products/wesley/gribw.html

– regional subset of Lambert conformal grib files

URLs

● NARR: http://wwwt.emc.ncep.noaa.gov/mmb/rreanl/index.html

– home page for NARR ● NARR rotating archive: http://nomad7.ncep.noaa.gov/ncep_data ● NCAR Command Language (NCL): http://ngwww.ucar.edu/ – graphics, computations package that supports grib ● NCDC data: http://nomads.ncdc.noaa.gov/#narr_datasets – on-line supplier of NARR data ● wgrib: http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html

– inventory, slice & dice grib files