tl;dr There appears to be a bug in NetCDF Java with respect to reading linked
HDF4 files which results in data that is read from the linked file(s) to be
erroneous.
Resources
* ToolsUI
* HDFView
* The files mentioned below can be retrieved directly from
OceanColor<http://oceancolor.gsfc.nasa.gov/cgi/l3/A20021822013212.L3b_MC_RRS.main.bz2?sub=bin>
(one at a time), or (for convenience) as one tar.gz file
fromhere<https://drive.google.com/uc?id=0B6UT7Mn4GZQhMjdLNDBBMFE0TTA&export=download>
Details
I'm reading data from the Aqua MODIS L3 Binned products available from the NASA
OceanColor<http://oceancolor.gsfc.nasa.gov/> website. It should be noted that
these files are HDF4 (4.2.9 according to NetCDF Java - ncdump). Many of the
products, such as chlorophyll, Particulate Inorganic Carbon, and Sea Surface
Temperature, come as a single file. The NetCDF library reads these files
without any difficulty.
However, one of the datasets of interest is the Remote Sensing Reflectance
data, which is NOT provided as a single file, but as a "main" file and a set of
subordinate files which are read via the "main" file as needed (see here for
more information<http://oceancolor.gsfc.nasa.gov/PRODUCTS/modis_binned.html>):
* A20021822013212.L3b_MC_RRS.main
* A20021822013212.L3b_MC_RRS.x00
* A20021822013212.L3b_MC_RRS.x01
* A20021822013212.L3b_MC_RRS.x02
* A20021822013212.L3b_MC_RRS.x03
* A20021822013212.L3b_MC_RRS.x04
* A20021822013212.L3b_MC_RRS.x05
* A20021822013212.L3b_MC_RRS.x06
* A20021822013212.L3b_MC_RRS.x07
* A20021822013212.L3b_MC_RRS.x08
* A20021822013212.L3b_MC_RRS.x09
* A20021822013212.L3b_MC_RRS.x10
* A20021822013212.L3b_MC_RRS.x11
NetCDF Java (via ToolsUI) loads the .main file without issue, and permits
reading of data variables (i.e. Rrs_412) without raising any errors. However,
the data returned is not accurate. Below is a comparison of the data returned
by ToolsUI and the same data returned by HDFView (which uses the HDF-java
JNI<http://www.hdfgroup.org/products/java/JNI/> library):
Retrieving the first 10 values for variable "Rrs_412"
HDFView
Screen Capture<http://cl.ly/WZnD>
Opening the .main file in HDFView and looking at the Rrs_412 dataset gives a
very different set of data:
0.0055423053, 0.0106070135, 0.006894292, -0.0040368317, -0.0020879991,
-0.0020279996, 0.009794002, 0.011879213, 0.010874448, 0.012330733
ToolsUI
Screen Capture<http://cl.ly/WZMW>
Opening the .main file and performing an Ncdump Data of variable:
"Level-3_Binned_Data/Rrs_412(0:10:1).Rrs_412_sum"
Returns:
float Rrs_412_sum;
data:
{1.86057E-40, 9.403955E-38, 6.4099753E-10, 2.6076459E-9, 1.0297978E21,
5.6431478E-11, 0.0, -2.9699963E36, 4.59183E-40, 3.67343E-40, 2.60329423E11}
Also, in ToolsUI, all of the other data variables (i.e. angstrom, aot_869 &
Rrs_*) all display very very similar (most are identical) values as the
Rrs_412. This is not the case for HDFView.
Incidentally, reading the data via OceanColor's
SeaDas<http://seadas.gsfc.nasa.gov/> application (which uses NetCDF Java under
the hood) results in the same data as ToolsUI.
Wrap-up
The evidence above appears to indicate that there is a bug in NetCDF Java
related to linked HDF files which results in incorrect data reads from linked
files.
Does anyone have any idea:
a) what could be causing the issue?
b) how could it be addressed?
Thanks in advance,
Chris
