Transcript Selling a Product or Service

HDF4 OPeNDAP Project
Progress Report
MuQun Yang and Hyo-Kyung Lee
7/20/2015
HDF Developers' Meeting
1
OPeNDAP Review
• A software framework that allows
simple access to remote data
• Data Access Protocol
• Client-server model
• Widely used by Earth Science
Community
7/20/2015
HDF Developers' Meeting
2
OPeNDAP Reivew
Users
IDV
Hyrax
libnc-dap
AIRS
DAP
Visualization
Tools
OPeNDAP
Clients
OPeNDAP
Servers
HDF4
Files
OPeNDAP
Concept
Typical
Remote
Access
View Data
(User)
FTP/HTTP
Remote Data
(HDF4)
Visualization
Tools
tools
(e.g.
(gradsdap)
grads)
Handler
(hdf4_handler)
Client Library
(libnc-dap)
Server
(apache)
DAP Protocol (via http)
7/20/2015
HDF Developers' Meeting
4
Goals
Better Access for HDF-EOS2 via DAP
• Faster download through selection
• Quicker analysis through visualization
7/20/2015
HDF Developers' Meeting
5
HDTV Problems
You bought a HDTV but...
• No HD signal
• Needs more Channels in HD
• Cable Service Provider Problem
7/20/2015
HDF Developers' Meeting
6
HDF4 Problems
You downloaded a viewer but...
• No CF convention
• Needs more HDF4 data in CF
• Data Service Provider Problem
7/20/2015
HDF Developers' Meeting
7
Challenges: HDF-EOS2
HD Signal Challenge
• CF convention (e.g. NetCDF model)
• Syntax
• Clients expect a certain data structure
• Semantics
• Clients expect some key attributes
7/20/2015
HDF Developers' Meeting
8
Challenges: HDF-EOS2
More Channels in HD Challenge
• DVD/VHS/VCD/LD >>> Blue-Ray
• Can all titles be delivered in HD?
• HDF4/HDF-EOS2 >>> HDF5/HDF-EOS5
• Can all HDF-EOS2 be delivered in CF?
7/20/2015
HDF Developers' Meeting
9
Solution: New HDF4 Handler
HD Signal Solution
• Syntax
 Generate the correct DAP data type.
 Add shared geo-location information.
• Semantics
 Insert additional key attributes.
7/20/2015
HDF Developers' Meeting
10
Solution: New HDF4 Handler
More Channels in HD Solution
• Use Metadata parser
• Pro: simple
• Con: limited channels
• Use HDF-EOS2 library
• Pro: maximum channels
• Con: install HE2 library first
7/20/2015
HDF Developers' Meeting
11
Solution Comparison
7/20/2015
HDF Developers' Meeting
12
Quick Implementation
• Parse Metadata – hdf5_handler
• Use HDF-EOS2 library – h4toh5
7/20/2015
HDF Developers' Meeting
13
Current Status
• MODIS / AIRS Grid and Swath
• Other Projections (Polar, Sinusoidal)
• Testing
7/20/2015
HDF Developers' Meeting
14
HDF-EOS2 in CF via DAP
• Finally, Happy Data Consumer!
AIRS Grid
AIRS
Swath
MODIS
MODIS
Grid
Swath
AIRS.2008.10.27.L3.RetStd001.v5.2.2.0.G08303124144.hdf
ftp://airspar1u.ecs.nasa.gov/data/s4pa//Aqua_AIRS_Level2/AIRX2RET.005//2008/330/AIRS.2008.11.25.143.L2.RetStd.v5.2.2.0.G08
http://daac.gsfc.nasa.gov/daac-bin/whom/mk_page_cgi.pl?PATH=datapool/AIRS/Level3_V005/01_AIRX3STD/2008/10
MOD08_D3.A2008284.005.2008289105354.hdf
MOD06_L2.A2001222.0025.004.2003127111841.hdf
331121849.hdf
15
http://modis-atmos.gsfc.nasa.gov/MOD08_D3/acquiring.html
7/20/2015
HDF Developers' Meeting
http://modis-atmos.gsfc.nasa.gov/MOD06_L2/acquiring.html
Limitations – Missing Attributes
• MODIS Grid – Rain fall
• AURA OMI Grid - Ozone
No Unit!
7/20/2015
HDF Developers' Meeting
16
Limitations – Missing Names
• How to Match Dimension Names?
HDF4
Handler
pressureStd → Level
NetCDF4
Handler
7/20/2015
HDF Developers' Meeting
17
Limitations – Missing Values
• How to Supply Missing Dimension
Values?
MISR_AM1_TC_CLASSIFIERS_P020_O000995_F06_0011.hdf
HTTP/1.0 200 OK
XDODS-Server: 3.7.9
XOPeNDAP-Server: 3.7.9
XDAP: 3.1
Content-Type: text/plain
Content-Description: dods_error
Cache-Control: no-cache
Error {
code = 1002;
message = "An internal error was encountered:
This HDF-EOS2 file has multiple Grids with different dimension
sizes.
Please report this to [email protected]
";
};
7/20/2015
HDF Developers' Meeting
18
Solution – Customization
• Provide an option for handler
customization to generate fully CFcompliant output
• Create a separate XML input file for
handler and feed it into the handler.
• Insert attributes, renaming rules, and
missing values by putting them in the
XML input.
• This requires a knowledge about the
characteristics of data in advance.
7/20/2015
HDF Developers' Meeting
19
Solution – Customization
• Example: Attributes, Renaming, Values
<xml>
<variable name=‘RrLandRain’>
<attr name=‘unit’ type=‘String’> inches</attr>
<attr name=‘_FillValue’ type=‘Int16’>-1</attr>
</variable>
<rename from=‘GeoXTrack’ to=‘lon’ ></rename>
<rename from=‘pressStd’ to=‘lev’></rename>
<insert name=‘NAltitudeDim’ size=10 type=‘Int8’>
0,1,2,3,4,5,6,7,8,9
</insert>
</xml>
7/20/2015
HDF Developers' Meeting
20
Summary
• HDF4 Access via OPeNDAP
 Is Easy, Efficient and Fast
 Is Cool like HDTV
 Adding CF is not straightforward
• Use HDF-EOS2 library
 Is Essential
 Maximize the data coverage
7/20/2015
HDF Developers' Meeting
21
Future Work (Short Term)
• Expand test-suite with data pool
• Release through OPeNDAP website
• Support for Swath via Parsing
• Support for Point data type via HDFEOS2 library
7/20/2015
HDF Developers' Meeting
22
Future Work (Long Term)
• General HDF4 Files Support
• Customization for CF option
• HE2 to DAP Mapping Documentation
7/20/2015
HDF Developers' Meeting
23