Transcript incoming_grad_lectur..

Slides based on lectures from the
NCL workshop
http://www.ncl.ucar.edu/Training/Workshops/lectures.shtm
Also see the NCL homepage
http://www.ncl.ucar.edu/
This ppt can be downloaded off the
web at the below address
http://www.atmos.albany.edu/student/janiga/temp/incoming
_grad_lecture.ppt
[email protected]
For additional help…
http://www.ncl.ucar.edu/Support/ncl_talk.shtml
[[email protected] , [email protected]]
The NCL variable model is based on
netCDF
netCDF files
self describing
(ideally) all info contained within the file
– no external information needed to determine file contents
portable [machine independent]
–
Supported by many software tools / languages
–
–
–
NCL, IDL, Matlab, ferret, GrADS, F, C, C++,Java
command line file operators: NCO, CDO
 perform various tasks (i.e. concatenation) very efficiently
 http://nco.sourceforge.net
 http://www.mpimet.mpg.de/fileadmin/software/cdo
ncview
 quick visualization using gui if in COARDS format:
 IDV, NCVIEW, Panopoly, etc.
Similar to HDF (NASA) and GRIB (NCEP, ECMWF, etc) formats;
all these are read by NCL using the same one line.
Examining a netCDF file
•
ncdump file_name
–
•
ncdump -h file_name
–
–
•
dumps the entire contents of a file; prints every value
Dumps header information [most commonly used]
NCL equivalent: ncl_filedump file_name
ncdump -v U file_name
–
NCL equivalent: ncl_filedump –v U file_name
• ncl_filedump file_name
•
[ more general ]
– netCDF3/4, GRIB-1,GRIB-2, HDF, HDF-EOS [HDF5]
ncview file_name
[visualize file contents]
Now for an example….
Parts of netCDF file
DIMENSION SIZES
dimensions:
lat = 64
lon = 128
time = 12
FILE ATTRIBUTES
global attributes:
title = “Temp: 1999”
source = “NCAR”
conventions =
"None”
VARIABLES:
Names , Types, Attributes,
Coordinate Variables
variables:
float lat(lat)
lat:long_name = "latitude"
lat:units = "degrees_north"
float lon(lon)
lon:long_name = "longitude"
lon:units = "degrees_east"
int time(time)
time:long_name = "time"
time:units = "Month of Year"
double T(time, lat, lon)
T:long_name = “Temperature”
T:units = “C"
T:missing_value = 1.e+20f
T:_FillValue = 1.e+20f
visual: simple 2D netCDF Variable
Latitude coordinate variable (1D, &)
coordinate variables (rectilinear grid)
attributes @:
• long_name
• units
Longitude coordinate variable (1D, &)
Non-Traditional Grids or Arrays
•
netCDF and NCL can handle various “other” types of data:
– Spectral coefficents (YOTC)
– Lambert conformal (WRF)
http://www.ncl.ucar.edu/Applications/wrflc.shtml
– Radiosondes
– Surface station data
– Hurricane tracks
http://www.ncl.ucar.edu/Applications/unique.shtml
– Make your own!
•
Tons of pre-defined functions for manipulating and
converting between different formats as well as plotting.
http://www.ncl.ucar.edu/Document/Functions/list_alpha.shtm
netCDF [NCL] Variable model
X
Scalar
or
Array
attributes
long_name
_FillValue
units
add_offset
scale_factor
etc.
f = addfile(“foo.nc”,”r”)
x = f->X
; grb/hdf
NCL reads the scalar/array,
attributes, and coordinate
variables as an object
X
accessed via @
values
coordinates
time
lev
lat
lon
etc.
Scalar
or
Array
accessed via &
attributes
long_name
_FillValue
units
add_offset
scale_factor
etc.
coord var
time
lev
lat
lon
etc.
Detailed Look netCDF Variable (NCL)
ncl <return>
; interactive mode
ncl 0 > f = addfile ("UV300.nc", "r")
; open file
ncl 1 > u = f->U
; import STRUCTURE
ncl 2 > printVarSummary (u)
; overview of variable
Variable: u
Type: float
Total Size: 65536 bytes
16384 values
Number of Dimensions: 3
Dimensions and Sizes: [time | 2] x [lat | 64] x [lon | 128]
Coordinates:
time: [ 1 .. 7 ]
lat: [ -87.8638 .. 87.8638 ]
lon: [ 0 .. 357.185]
Number of Attributes: 5
_FillValue :
1e36
units :
m/s
long_name :
Zonal Wind Component
short_name :
U
missing_value : 1e36
Classic netCDF
Variable Model
NCL
syntax/funcs
query
use
modify
add
any aspect of
data object
•
NCAR Command Language
Complete Programming Language
data types
– variables
– operators
– expressions
– conditional statements
– loops
– functions/procedures/graphics
–
•
Features
query / manipulate meta data
– import data in a variety of formats
– array syntax / operations
– can use user fortran/C codes and commercial libraries
– most functions/procedures ignore missing data
–
Running NCL
•
Interactive Mode (Command line)
–
–
•
ncl [options][command-line-arguments] <return>
ncl> enter commands
ncl> quit
<return>
can save interactive commands
ncl> record “file_name”
ncl> stop record
Batch Mode
[ .ncl suffix is optional]
ncl [options][arguments] script.ncl
– ncl < script.ncl
[also acceptable]
– nohup ncl [options][arguments] script.ncl >
out.txt &
–
–"&"
means put in background
–“nohup” means don’t quit when I logoff (ie long scripts).
NCL predefined options
•
•
•
ncl –hnxV [predfined options are preceded by dash]
may be used for interactive or batch mode
informational
–
–
•
action
–
–
•
ncl –h [display predefined options and usage and exit]
ncl –V [print the NCL version and exit]
ncl –x [echo statements as encountered (debug)]
ncl –n [don't enumerate dimensions of values in print() ]
multiple predefined options
–
ncl –nx
[ not ncl –n –x ]
Variables can be set on the command line prior to a script.
However, this tends to not be necessary.
Input from a text file or systemfunc()
http://www.ncl.ucar.edu/Document/Manuals/Ref_Manual/NclCL
O.shtml
Sample Batch Script:
sample.ncl
load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_code.ncl“
load “$NCARG_ROOT/lib/ncarg/nclscripts/csm/gsn_csm.ncl“
begin
; optional
f1 = addfile("TMP_58-97.nc", "r")
; open netCDF file
T
= f1->Tmp
; T(time,lev,lat,lon) =>(480,17,73,144)
f2 = addfile(“P_1958-1997.grb", "r") ; open GRIB file
P
= f2->Pres
; P(time,lev,lat,lon)
f3 = addfile(“Q_1958-1997.hdfeos", "r") ; open hdf –eos file
Q
= f3->Specific_Humidity
; Q(time,lev,lat,lon)
pot = T* (1.+0.622*Q) *(1000/P)^0.286
; potential temperature (array)
print("pot: min="+ min(pot) + " max="+ max(pot) )
wks = gsn_open_wks("ps", “sample")
; open a graphic workstation
gsn_define_colormap (wks,"gui_default") ; change from default
plot = gsn_csm_contour_map_polar (wks,T(0,5,:,:),False)
res = True
; change default plot
res@cnFillOn
= True
; use colors
res@gsnSpreadColors = True
; use entire color map
plot = gsn_csm_pres_hgt (wks,T(0,:,{50},:), res )
end
; only if begin is present
Outline: NCL Language Basics
•
•
•
•
•
•
•
•
•
•
special syntax characters
data types
Variables  netCDF/NCL variable model
attributes
_FillValue
named dimensions
coordinate variables
print and printVarSummary
shaping
subscripting
NCL Syntax Characters
; - comment [can appear anywhere]
• @ - reference/create attributes
• ! - reference/create named dimension
• & - reference/create coordinate variable
•{…} - coordinate subscripting
• $ - enclose strings when (im/ex)port variables via addfile
•(/../) - array construct characters
• : - array syntax
• | - separator for named dimensions
• \ - continue character [statement to span multiple lines]
• :: - syntax for external shared objects (eg, fortran/C)
• -> - use to (im/ex)port variables via addfile function
•
•“
“
- use for a string
Data Types
Numeric
• double (64 bit)
• float
(32 bit)
• long
(32 or 64 bit)
• integer (32 bit)
• short
(16 bit)
• byte
( 8 bit, 0-255)
• complex NOT supported
non-Numeric
• string
• character
• graphic
• file
• logical
• list
Time coordinate variables frequently double.
If IR data (K) was offset by 100 it could be stored in byte instead of double (8
times less disk space) or float (4 times less disk space).
Data will often be packed into short or byte this way xFloat = short2flt(xShort)
xShort@scale_factor
xShort@add_offset
Attributes
•
info about a variable or file [meta data]
attributes can be any data type except file
– scalar, multi dimensional array (string, numeric)
–
•
assign/access with @ character
T@units
= “deg C”
– T@long_name = “temperature”
– T@whatever = an array of any dimension or type
– T@_FillValue = -999.
– plot title uses x@long_name by default
–
•
attribute functions [isatt, getfilevaratts]
–
•
if ([email protected].”K”) then
delete can eliminate an attribute
–
delete(T@title)
_FillValue
•
Unidata and NCL reserved attribute
most NCL functions ignore _FillValue
•
Note: “missing_value” attribute: no special status to NCL
•
–
–
if “T” has “missing_value” attribute but no “_FillValue ”
 NCL creates: T@_FillValue = T@missing_value
use built-in function “ismissing” to check for _FillValue
 if (any(ismissing(T))) then
 T = where (ismissing(T),0,T)
• netCDF Operators [NCO]: _FillValue attribute
• ncview: missing_value attribute
–
best to create netCDF files with both
• NCL: don’t use commonly encountered values
as a _FillValue!
Arrays
•
row major ….. like C/C++
 left dimension varies slowest; right varies fastest
 dimension numbering left to right [0,1,..]
•
indicies [subscripts] are zero based [0,N-1]
Consider T(:,:,:)
left
dimension is 0
middle dimension is 1
right
dimension is 2
[ T!0 ]
[ T!1 ]
[ T!2 ]
NCL (netCDF) Dimensions
• may be “named”
–provides
alternative way to reference subscripts
–recommendation: always name dimensions
–use NCL syntax
• assigned with ! character
{let T(:,:,:)}
–
T!0 = "time"
–
T!1 = "lat“
–
T!2 = "lon"
; leftmost [slowest varying] dim
; rightmost [fastest varying] dim
dim names may be renamed, retrieved
– T!1 = "LAT" … dName = T!2
• delete can eliminate: delete (T!2)
•
•
Named dimensions used to reshape
NCL (netCDF) Coordinate Variables
•
data coordinates [ eg: time, level, lat, lon ]
–
–
strict netCDF definition
 1D numeric array
 monotonically in[de]creasing numeric values
 can only be assigned to a named dimension
 1D array must be the same size as dimension
 should not have _FillValue attribute
coordinate values assigned via & character
 T&lat = latitude ; Here latitude is a 1D array
• used in graphics, coordinate subscripting
• coordinate functions
iscoord , isfilevarcoord
• delete can eliminate coordinate array
–delete(T&time)
–
Create and Assign Coordinate Variables
•
create 1D array
time = (/ 1980, 1981, 1982 /)
– lon = (/ -90, ... 90 /)
; integer
; float
–
•
assign dimension name
[same as variable name]
time!0 = “time”
– lon!0 = “lon”
–
•
assign values to named dimension
time&time = time
– lon&lon
= lon
–
•
let x be 2D: name dimensions
–
•
x!0 = “time”
… x!1 = “lon”
assign coordinate variables to x
–
x&time = time … x&lon = lon
Access/Change/Create/Delete Meta Data
•
@
attributes
u@long_name = "U"
– lonName = u@long_name
–
•
!
named dimensions
u!0 = "TIME"
– tName = u!0
–
•
&
coordinate variable
u&lat = (/ -90., -85, .... , 85., 90. /)
– latitude = u&lat
–
•
$
substitute string
–
x = fin->$variable(n)$(date,{-30:60},:)
Variable Assignment
•
Variable-to-Variable assignment
–
consider y = x where x is previously defined
if y not defined:
 y has same type/shape/values/meta data as x
 if y predefined:
 y must have same shape and type
 or, x must be coerceible to the type of y
 y attributes, dimension names and coordinate
variables, if they exist, will be over written

•
Value-only assignment (no meta copy)
–
U multi-dimensional array with meta data
Uval = (/ U /) or Uval = (/ f->U/)
 the (/ ... /) operator pair strips meta data

Variable Subscripting (1 of 3)
Standard Array Subscripting
•
•
•
ranges: start/end and [optional] stride
indices separated by :
omitting start/end index implies default begin/end
Consider T(time,lat,lon)
T
T(0,:,::5)
T(0,::-1,:50)
T(:1,45,10:20)



entire array [ don't use T(:,:,:) ]
1st time index, all lat, every 5th lon
1st time index, reverse lat order, 1st
51 lon

1st 2 time indices, 46th index of lat,
10-20 indicies of lon
Variable Subscripting
(2 of 3)
Coordinate Variable Subscripting
•
•
•
•
only applies to coordinate variables
same rules apply for ranges, strides, defaults
use curly brackets {…}
standard and coordinate subs can be mixed
[if no reorder]
 all times/lon, lat -30° to +30°
(inclusive)
T(0,{-20},{-180:35:3})  1st time, lat nearest - 20°, every
3rd lon between -180° and
35°
T(:,{-30:30},:)
Variable Subscripting
(3 of 3)
Named Dimensions
• only used for dimension reordering
• indicated by |
• dim names must be used for each subscript
• named/coordinate subscripting can be mixed
Consider T(time,lat,lon)
t = T(lat|:, lon|:, time|:)
 makes t(lat,lon,time)
t = T(time|:,{lon|90:120},{lat|-20:20})  all times,
90-120° lon, -20-20°
lat
Standard and Coordinate Subscripting
Latitude coordinate variable (1D)
Standard:
T(9:13,1:8)
Coordinate:
T({-10:20},{-170:-110})
Combined:
Longitude coordinate variable (1D)
T({-10:20}, 1:8)
“printing”
•
printVarSummary
–
•
gives gross overview of a variable
print
same info as printVarSummary
– prints values
–
•
write_matrix
print to standard out or a file
– format control of numerical output
– can write to file also
–
print (1 of 2)
•
Prints out all variable information including
–
–
meta data, values
T(lat,lon): print (T)
Variable: T
Type: float
Total Size: 32768 bytes
8192 values
Number of Dimensions: 2
Dimensions / Sizes: [lat | 64] x [lon | 128]
Coordinates:
lat:
[-87.86379 .. 87.86379]
lon: [ 0. 0 .. 357.1875]
Number of Attributes: 2
long_name: Temperature
units:
C
(0,0) -31.7
(0,1) -31.4
(0,2) -32.3
(0,3) -33.4
(0,4) -31.3 etc. [entire T array will be printed]
print (2 of 2)
•
can be used to print a subset of array
–
–
meta data, values
T(lat,lon): print( T(:,103) ) or print( T(:,{110}) )
Variable: T (subsection)
Type: float
Total Size: 256 bytes
64 values
Number of Dimensions: 1
Dimensions / Sizes: [lat | 64]
Coordinates:
lat:
[-87.86379 .. 87.86379]
Number of Attributes: 3
long_name: Temperature
units:
C
lon:
109.6875 [ added ]
(0) -40.7
(1) -33.0
(2) -25.1
(3) -20.0
(4) -15.3 etc.
Things that can be done (look up or
ask)
•
•
•
•
•
•
Write netCDF files (compression, making large files)
Read and write binary and ascii
Functions and procedures in NCL language (over 1000),
fortran or C (most are earth science orientated; essentially
Matlab or IDV for earth science).
Loops, conditional statements.
OPeNDAP support (query and read files directly from the
web without downloading the whole file)
http://www.ncl.ucar.edu/Document/Function
s/index.shtml