Transcript PIRATA data and products available through a DODS/OPeNDAP Python server Why PIRATA?

PIRATA data and products available through a
DODS/OPeNDAP Python server
Guilherme P. Castelão & Roberto A. F. De Almeida
●
Why PIRATA?
●
Why DODS/OPeNDAP?
●
Why Python?
●
Future work.
Guilherme P. Castelão <[email protected]>
Roberto A. F. De Almeida <[email protected]>
http://opendap.oceanografia.org
Why PIRATA?
Or better: what is PIRATA?
●
●
●
●
●
PIlot Research moored Array in the Tropical Atlantic (PIRATA) maintains a
set of 12 ATLAS moorings in the Tropical Atlantic.
Sensors measure sub-surface temperature and salinity from 1 to 500 m
depth, near surface temperature and humidity of the air, incident short
wave radiation, direction and magnitude of the wind and rainfall.
Deployments started on late 1997, composing a daily mean time series that
covers now more than 6 years on most positions.
Ocean surface heat fluxes can be estimated by bulk formulae, whose
dependencies are satisfied by the PIRATA dataset.
There is no precedent dataset with such characteristics at the considered
region, which makes possible studies unfeasible until now.
PIRATA data and products available through a DODS/OPeNDAP Python server – 2/9
Why PIRATA?
Buoys' locations
PIRATA data and products available through a DODS/OPeNDAP Python server – 3/9
Why DODS/OPeNDAP?
If PIRATA is the question, DODS is the answer.
●
●
●
PIRATA data has been made available only on the Pacific Marine
Environmental Laboratory (PMEL) web site through electronic forms.
It is an easy process to download one or a few datasets, but it's not
efficient for more frequent or automatic updates.
DODS/OPeNDAP solves this problem in a way that is easy and transparent
to the user. In the best-case scenarios, no additional software or training is
necessary.
PIRATA data and products available through a DODS/OPeNDAP Python server – 4/9
Why DODS/OPeNDAP?
What is DODS/OPeNDAP?
●
●
●
●
Fall 1993 - Initial workshop held at the URI to discuss the design and
construction of a system that would:
●
facilitate access to oceanographic datasets
●
allow users to use their preferred analysis package
Project funded by NASA, NOAA, NOPP, with 23 partners.
DODS is an architectural framework to allow the user to easily access data
over the network in a consistent fashion. DODS provides a basic structure
to move data over the network.
OPeNDAP (Open source Project for a Network Data Access Protocol) is a
non-profit corporation established to develop and promote software that
facilitates access to data via the network.
PIRATA data and products available through a DODS/OPeNDAP Python server – 5/9
Why DODS/OPeNDAP?
What does this mean in practice?
●
Users can access data transparently through the network.
●
Ferret, e.g., is a DODS client:
set data “sst.nc”
●
set data “http://opendap.oceanografia.org/data/sst.nc“
●
●
●
●
Data is downloaded automatically only for the sections in space-time that
are actually used, being bandwidth and memory efficient.
There are plenty of DODS clients: Ferret, GrADS, IDL, Matlab, ncview, ...,
and libraries available for C++, Java and Python.
DODS servers support a variety of data formats: NetCDF, HDF, GrADS,
Matlab, etc. Data format doesn't matter for the client!
PIRATA data and products available through a DODS/OPeNDAP Python server – 6/9
Why DODS/OPeNDAP?
The PIRATA DODS server.
●
●
●
We developed a DODS server using the Python scripting language. The
server has initial support for arbitrary NetCDF datasets, and is easily
extendable through plugins.
PIRATA data is downloaded automatically from the PMEL web site and
converted to NetCDF. Heat fluxes are then estimated from the data, and
both are made available on the DODS server.
Initially only data for the buoy at 0°23°W is being processed:
http://opendap.oceanografia.org/data/pirata_data_0023.nc
●
http://opendap.oceanografia.org/data/pirata_flux_0023.nc
●
PIRATA data and products available through a DODS/OPeNDAP Python server – 7/9
Why Python?
Simple is better than complex.
●
●
●
●
Python is a free, open-source, general-purpose, high-level, interpreted,
object-oriented, introspective, extensible, dynamic, robust, cross-plataform,
collaborative programming language.
It was also created by a dutch, Guido van Rossum, who named it after the
Monty Python's Flying Circus.
Python use for science is gradually increasing. It's easy to learn, fast to
program and deploy language. It has a matricial framework similar to
Matlab.
CDAT (Climate Data Analysis Tools) and PyClimate are two nice examples of
Python being used for scientifics, climate related, applications.
PIRATA data and products available through a DODS/OPeNDAP Python server – 8/9
Future Work
How can the server be improved?
●
●
The obvious next step is to process the data from all 12 buoys, and make it
available through the DODS server.
The intermediate step of storing the data and products as NetCDF is not
really necessary, and adds to the already costy processing time. A planned
solution is to store and serve the data using a free relational database
(PostgreSQL).
PIRATA data and products available through a DODS/OPeNDAP Python server – 9/9