Transcript Document 7726650
NARR User Issues
Wesley Ebisuzaki CPC/NCEP [email protected]
http:wwwt.emc.ncep.noaa.gov/mmb/rreanl/
Topics
● Files: contents, availability, formats ● Tools: BUFR, GRIB ● Obtaining the data: NCEP, NCAR, NCDC ● Scripting downloads ● What is reality? Is reanalysis truth? An insider's viewpoint.
GRIB Files
● For grids (horizontal fields) such as 500 mb Z ● Lambert conformal grid: 32 km eta grid (#221) – Have grids of lat/lon, display programs ● Model grid: not available except at NCEP ● – No plans to make available, display programs?
Analyses are on-line and forecasts ?
● 99% of people will use the GRIB files
BUFR
● For all input observations except SST, snow, ice, observed precipitation (GRIB) ● Input obs BUFR files have QC marks ● Analyses/forecasts at station locations (MOLTS) – Much smaller than GRIB files, limited locations ● Distribution (informal) – Includes restricted data – Few users (experience from global reanalysis) – Often other sources of data (in easier formats)
Misc Files
● Restart files: huge, full precision, NCEP-only ● Statistics: – Fit of obs to NARR, fit of obs to R2 ● – Small, can be made available on-line Observation counts: – Small, should be put on-line ● PNG (monitoring plots) – Small, should be put on-line
Want more files?
Don't expect too much.
NARR has zero funding.
Despair?
● Global Reanalysis has been running this way for years!
● CPC has been running global reanalysis for its own operations. CPC and the outside community have shared interest in maintaining a stable assimilation system. NCEP has an interest in others using its products.
● However, limited resources, need to be selective, work with partners
Status
● Real-time is running (4 days late) ● Started building 1.5-2 year on-line rotating archive of latest analyses at NCEP ● 1979-2003 merged files at NCAR, NCDC ● NCDC has 5 TB merged files on-line, trying to support the on-line community by NOMADS ● EMC people are working on other projects
Tools: BUFR
● Situation improving ● GrADS can read bufr directly
Tools: GRIB
● GrADS: display and computations – grib2ctl.pl: make GrADS control files ● wgrib: inventory, decode GRIB files, remove records from a file (save space) ● lcgrib: regional subset (save space) ● copygb: change grid (save space or neurons) ● grbpoint.pl: grib -> point value printout (slow) ● Gempak (?), NCL (?)
Example: GrADS
● Make control file (analyses) – grib2ctl.pl -rr cpc_rr_mon.197901 >my.ctl
– grib2ctl.pl -rr cpc_rr_mon.%y4%m2 >my.ctl
● Make index file (analyses) ● – gribmap -0 -i my.ctl
Run GrADS – grads – open my.ctl
Example: wgrib
● Use: inventories, make files smaller, decode data for fortran and C programs ● See contents of a grib file – wgrib (options) cpc_mon_197901 ● To extract records, decode records, see wgrib home page.
Example: lcgrib
● Regional subset of Lambert conformal grid ● *** MAKE SMALLER FILES *** ● lcgrib in-grib out-grib west south east north – West, south, east and north are in degrees – N and E are +ve, S and W are -ve
Obtaining Data: NCEP
● Rotating archive: latest 1.5 – 2 years of analyses ● – http://nomad7.ncep.noaa.gov/ncep_data Other files: TBD ● NCEP not set up to deal with the public requests – Go through the web – Go through NCDC, NCAR
Obtaining Data: NCAR
● Have 1979-2003 merged data ● Sure they will get more of the data ● Plans, availability?
Obtaining Data: NCDC
● NCDC has the merged data 1979-2003 and want 2004 ASAP ● NCDC will use NOMADS to distribute the data to the on-line community.
● NOMADS is project of many institutions including NCDC and NCEP for distributing data over the internet.
● NOMADS-NCDC is taking software from NCEP and COLA to distribute GRIB data.
Data Access at NCDC
● Custom plots ● ftp4u: access by selected time, field, region ● http access to raw data (like FTP) ● GDS (GrADS Data Server): OpeNDAP server (formerly DODS), on-line access to data
Difficulties in on-line distribution
● Downloading a 5 TB data set is difficult ● Need ways to minimize data transfer ● Plots (maps) work but limited utility ● GDS needs to be optimized for large datasets ● ftp4u is a solution
FTP4U
● Only download data that you need – Select levels – Select variables – Select geographic regions – Select times
Making FTP4U Work
● Don't request more data than needed ● Make one big request, not many small requests of the same file; i.e., make one request to get the winds, temp and heights rather than 3 requests. Do not request long time series.
● Shared resource (1 job at a time) ● Be prepared to script/automate the downloads – May be impossible to download in a few hours, may require days, weeks and multiple jobs/requests
Scripting FTP4U Downloads
● FTP4U is normally accessed by a browser ● Anything a browser can do, a script can also do ● Experience at NCEP is that most plot/ftp4u access is scripted (automated) ● NCDC's modifications to the NCEP provided programs make it more difficult to script the download but nothing that a cURL user couldn't handle. NCDC wants to change!
Scripting Downloads cont'd
● Use cURL, a unix/windows program, to “emulate” a browser ● Each browser page becomes a call to cURL.
● With a little RTFM was able get script working ● sample program will be put on NARR home page
Scripting Downloads cont'd
● Downside: steep learning curve, may have no local experts for help ● Upside: can make large data transfers possible, can be used in other projects (ex. getting fcst data automatically, skewing on line opinion polls) ● Given that we get more data off the web, it's a skill that someone in a large group should eventually have.
Obtaining Data: NCDC
● It's a learning situation for NCDC ● Give feedback (complain), it's the only way to improve the situation. Lessons learned here will help the next time you try getting a different dataset from NCDC's nomad server.
● Current issues: full disk (new RAID is will be installed soon), obtaining time series, scripting support
What is Reality?
● Some people consider reanalysis to be truth ● NARR is new, bound to be warts, already found some.
What is Reality?
● Accuracy of reanalysis depends on many factors – Data going into the analyses ● Types, counts, quality of observations – Accuracy of previous analysis – Accuracy of atmospheric/land model – Spatial and temporal filtering – NO EASY ANSWER
What is Reality? cont'd
● Accuracy – check to the appropriate data counts and biases ● Need to add our counts to web ● Need to add fit to data to web – check other reanalyses – Fit to observations – Consider sensitivity to modeling assumptions – Sensitivity experiments (data, resolution, model), only available to limited # of people
What is Reality? cont'd “Trust but verify”
Ronald Reagan
URLs
● copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb – change grid type of grib files ● copygb: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/copygb – change grid type of grib files ● cURL: http://curl.haxx.se
– download URLs, handles post data ● GEMPAK: http://my.unidata.ucar.edu/content/software/gempak/index.html
– graphics, can handle NARR files with some effort ● GrADS: http://grads.iges.org/grads/ – graphics, computations
URLs
● grbpoint.pl: ftp://ftpprd.ncep.noaa.gov/pub/cpc/wd51we/wgrib.scripts
– simple program to extract data from lat/lon ● grib2ctl.pl: http://www.cpc.ncep.noaa.gov/products/wesley/grib2ctl.html
– makes control files for GrADS ● g95: http://www.g95.org/ – f95 compiler for copygb ● HIPHOP: http://iacweb.etzh.ch/stall/dominik/hihop/ – grib support for idl ● lcgrib: http://www.cpc.ncep.noaa.gov/products/wesley/gribw.html
– regional subset of Lambert conformal grib files
URLs
● NARR: http://wwwt.emc.ncep.noaa.gov/mmb/rreanl/index.html
– home page for NARR ● NARR rotating archive: http://nomad7.ncep.noaa.gov/ncep_data ● NCAR Command Language (NCL): http://ngwww.ucar.edu/ – graphics, computations package that supports grib ● NCDC data: http://nomads.ncdc.noaa.gov/#narr_datasets – on-line supplier of NARR data ● wgrib: http://www.cpc.ncep.noaa.gov/products/wesley/wgrib.html
– inventory, slice & dice grib files