Transcript www.opendap.org

The Server-Side with FTDS
Roland Schweitzer
Ansley Manke, Kevin O’Brien, Karl Smith and
Steve Hankin
The Ferret-THREDDS Data Server
● CDM IOSP and plug-in to TDS that allows
Ferret to act on data being served.
● Uses delayed evaluation:
➢ Transformation applied virtually over the
entire domain
➢ Computations done as requested and only
over the required sub-domain
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Delayed Evaluation
Global 2-degree
with 12 time steps.
Client requests an area average
of SST over all time steps.
Server responds to a OPeNDAP
URL in which the transformed
variable and its
coordinate variable exists with
no computations performed.
Grid {
ARRAY:
Float32 SST[TIME = 12]
[COADSY = 90]
[COADSX = 180];
MAPS:
Float64 TIME[TIME = 12];
Float64 COADSY[COADSY = 90];
Float64 COADSX[COADSX = 180];
} SST;
Float32 sst_1_transformed[TIME = 12];
Delayed Evaluation Continued
F-TDS computes the area
average for the first three
time steps.
Client requests value of
the first three months.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
Client receives the
requested values and
matching coordinates.
OPeNDAP Workshop - July 7, 2014
The Ferret-THREDDS Data Server
● Two methods of use:
➢ Custom URLs
➢ Ferret scripts on the server
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Custom URL
● We just saw an example of a client
request using a special URL syntax to
request a computation.
● Sadly, it looks like this:
http://dunkel.pmel.noaa.gov:8920/thredds/dodsC/las/coads_climatology_cd
f/data_coads_climatology.jnl_expr_%7B%7D%7Bletdeq1+sst_1_transformed%3
Dsst%5Bd%3D1%2Cx%3D135%3A285%40Average%2Cy%3D30%3A30%40Average%5D_cr_ATTRCMD+sst+sst_1_transformed%7D.html
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Custom URL syntax
http://server.gov/thredds/dodsC/ (server)
coads.nc (untransformed data set)
_expr_{} (magic spell incantation)
{letdeq1 sst_1_transformed=
sst[d=1,x=135:285@ave,y=-30:30@ave]
_cr_
ATTRCMD sst sst_1_transformed} (Ferret)
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
More about that URL
First hard lesson learned:
1. Cramming Ferret syntax into a URL is a
near disaster.
Syntax elements like the “/” in let/d=1,
and the “;” to separate lines are also
significant to HTTP clients and servers.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
So what to do?
These difficulties make it hard for people
to spontaneously define transformations to
be performed on the server while working
in a client.
Teach the client to assist...
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
LAS
We’ve already seen a
snippet of one client
(the web-based Live
Access Server) which
offers a rich UI for
defining
transformations.
let/remote
● Our analysis client can build the custom
URL for you if indicate that the OPeNDAP
server is F-TDS ready.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
upper ocean average temperature
let Tave Z
=
1
ΔZ
∫
Z1
Temp dz
0
use “http://server/dataset.nc”
let/remote TaveZ = Temp[z=0:`z1`@ave]
plot TaveZ[X=180,Y=0]
fill/T=1-jan-2050
TaveZ
OPeNDAP Workshop - July 7, 2014
average eddy kinetic energy
let Ek
=
let Ekave
=
T
u2 + v2
1
ΔT
∫
T2
Ek dt
T2
use “http://server/dataset.nc”
let/remote Ek = u^2 + v^2
let/remote EkaveT = Ek[T=`T1`:`T2`@ave]
fill EkaveT[Z=0]
OPeNDAP Workshop - July 7, 2014
Custom scripts on the server
● Any Ferret command script that defines
new variables from an existing data source
can be used to create a virtual data set.
● Variables are evaluated in delayed-mode.
● Instantly provide access to many types of
analysis and transformations of existing
variables.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Custom scripts on the server
● Removes the need for the client to assist.
● User sees the original data set, plus the
transformed virtual variables.
● Scripting conventions determine which
variables in the calculation are hidden and
which variables appear in the final data set.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
A success story
ROMS - Regional Ocean Modeling System
● The challenge, take the curvilinear,
rotated and staggered grid for U and V
from the model and serve them via
OPeNDAP with it without the rotation
and stagger.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
The UV Model Grid
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Vectors from the original grid
● Don’t make much sense…
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
The Final Product
● U and V on a single grid without the
rotation
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
The original variables
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Plus the new rotated U and V
OPeNDAP Workshop - July 7, 2014
Applying more transformations
OPeNDAP Workshop - July 7, 2014
Discrete Sampling Geometry Files
● Difficult to work with over OPeNDAP due
to the large number of network requests.
● What if we could make “delayed-mode”
virtual sub-sets?
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Discrete Sampling Geometry File
● F-TDS has extended the concept of a
virtual variables to CF-1.7 Discrete
Sampling Geometry (DSG) files.
● Clients request sub-sets via projection
style queries and get back fully formed
DSG virtual data set.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Example: full dataset
> ncdump –h http://SERVER/geoide/dodsC/drifter/DrifterBuoyData_1_5000.nc
netcdf DrifterBuoyData_1_5000 {
dimensions:
TRAJ = 5000 ;
OBS = 8332804 ;
variables:
float DLAT(TRAJ) ;
DLAT:long_name = "Deployment latitude" ;
float DLON(TRAJ) ;
DLON:long_name = "Deployment longitude" ;
double TIME(OBS) ;
TIME:units = "seconds since 1-jan-1970" ;
float LAT(OBS) ;
LAT:long_name = "Latitude" ;
float LON(OBS) ;
LON:long_name = "Longitude" ;
float TEMP(OBS) ;
TEMP:long_name = "Temperature" ;
OPeNDAP Workshop - July 7, 2014
Make new virtual data set
baseURL + _expr_{}{constraint}{}
constraint is a Ferret expression
e.g. let mask = lat GT 0 and lat LT 20
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Subset dataset
> ncdump -h
http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/las/DrifterBuoyData_1_5000.nc_expr_%7B/home/data/kobrien/
Drifting_buoys.nc_cr_let%20mask%20=%20if%20%20dlat%20GT%200%20and%20dlat%20LT%2010%20then%201
%7D%7Bgo%20dg_subset.jnl%7D
netcdf
Drifting_buoys.nc_cr_let\%20mask\%20\=\%20if\%20\%20dlat\%20GT\%200\%20and\%20dlat\%20LT\%2010\%2
0then\%201\%7D\%7Bgo\%20dg_subset {
dimensions:
TRAJ = 1042;
OBS = 1481332 ;
variables:
float DLAT(TRAJ) ;
DLAT:long_name = "Deployment latitude" ;
float DLON(TRAJ) ;
DLON:long_name = "Deployment longitude" ;
double TIME(OBS) ;
TIME:units = "seconds since 1-jan-1970" ;
float LAT(OBS) ;
LAT:long_name = "Latitude" ;
float LON(OBS) ;
LON:long_name = "Longitude" ;
float TEMP(OBS) ;
TEMP:long_name = "Temperature" ;
OPeNDAP Workshop - July 7, 2014
Using the full dataset
> ferret
yes? use "http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/drifter/DrifterBuoyData_1_5000.nc"
yes? go basemap
yes? plot/vs/overlay dlon, dlat
OPeNDAP Workshop - July 7, 2014
Using the subset dataset
> ferret
yes? use
"http://dunkel.pmel.noaa.gov:8680/geoide/dodsC/las/DrifterBuoyData_1_5000.nc_expr
_%7B/home/data/kobrien/Drifting_buoys.nc_cr_let%20mask%20=%20if%20%20dlat%2
0GT%200%20and%20dlat%20LT%2010%20then%201%7D%7Bgo%20dg_subset.jnl%7D"
yes? go basemap
yes? plot/vs/overlay dlon, dlat
OPeNDAP Workshop - July 7, 2014
F-TDS & the OPeNDAP Community
●
●
●
●
F-TDS is a powerful tool.
F-TDS is not widely used.
Users need help from the client.
Complex transformation are possible
with specialized knowledge and best
done on the server by data providers.
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
F-TDS & the OPeNDAP Community
● Installation, integration, configuration
and maintenance are just difficult
enough to keep people from using the
software.
What to do about it?
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
First Attempt...
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Server-side functions WG
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Happy to announce the results…
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Second attempt…
● Proposed a set of core transformations:
➢ averaging of time, area, depth, and volume
➢ computing variance, extrema, integrals and
counts of valid/invalid points
➢ linear combinations (difference variables,
convert units)
➢ re-gridding to arbitrary lat/lon rectangular
girds
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Also proposed
● Simplified direct implementation to
make it deployable without Ferret
● General syntax (not Ferret script-based)
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
Community Effort
● No funding was awarded
● Any progress will be a community effort
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014
More Information
● [email protected]
● http://ferret.pmel.noaa.gov/LAS (search
F-TDS)
NOAA/OAR/PMEL, UW JISAO, and Weathertop Consulting, LLC
OPeNDAP Workshop - July 7, 2014