Hi John:
Thanks.
The L2 files we process and make our own composite files so they aren’t an
issue It is mainly the L3’s (SMI) that we are concerned about. As I said,
the two issues are just can they be served by TDS, forget any other problems
(and we can do what we have been doing is add time as a coordinate in an NcML
file which is what we then aggregate).
But the bigger issue is whether we can aggregate over the different file types.
If we are actually aggregating NcML files (which is usually the case), will
TDS care that some of the files being pointed to in the NcML are HDF4 and some
are netcdf4? If TDS doesn’t care that the filetypes are different, we
probably can kludge a solution to a lot of the issues I raised. Otherwise, I
may have to figure out how to re-download a ton of files over a bandwidth that
is already often over-saturated.
And can I open up the can of worms again? I am curious what people feel about
the use of groups in netcdf4 files? It is no longer a theoretical question.
If you are getting ocean color data from NASA, that is what you are getting.
If people are interested I can post the file headers (or maybe more
appropriately move this discussion to the netcdf mail-list) so you can see how
they are being used. If you are familiar with the old HDF4 format, it is
mostly a mapping of that structure.
-Roy
> On Jun 1, 2015, at 7:20 PM, John Caron <caron@xxxxxxxx> wrote:
>
> Hi Roy:
>
> The grids look ok (L3m), although they didnt put the time values into a
> coordinate, but in global attributes.
>
> The swaths (L2) can probably be fixed up with a little NcML. Its not clear if
> aggregation is possible or useful.
>
> The points (L3b) are incomprehensible to me. Someone expert would have to
> explain whats in there.
>
> Not a lot of CF compliance that I can see.
>
> John
>
> On Mon, Jun 1, 2015 at 5:24 PM, Roy Mendelssohn - NOAA Federal
> <roy.mendelssohn@xxxxxxxx> wrote:
> HI All:
>
> I don’t know how many of you get notices like this:
>
> > MODIS Ocean Color and SST Data Users:
> >
> > Within the next week, the Ocean Biology Processing Group will begin the
> > ocean color (OC) reprocessing of MODIS on Aqua (MODISA). At that time, we
> > will also make available the fully reprocessed SST from MODIS on Aqua and
> > Terra, and we will start producing forward-stream products in the
> > reprocessing configuration for all four data streams (MODIS SST and OC from
> > both satellite platforms). General information on the reprocessing changes
> > for OC and SST can be found here:
>
> The new format, if you don’t know, are netcdf-4 files, but they make heavy
> use of groups. As some of you may remember, a proposal to develop a standard
> for use of groups in CF was put forward by Charlie Zender and Ted Haberman,
> but got shot down. Be that as it may, the NASA product is what it is, and
> this raised some questions for our data services. To wit:
>
> 1. Does anyone know if TDS can handle these files correctly (because of the
> groups and not being strictly CF)? toolsUI seems to be able to read them
> okay, but I am not certain that it is picking up the data types properly (so
> recognizes that something is an array but not necessarily a grid)? Sample
> files are at:
>
> http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html
>
>
> 2. For some of the data we do some further processing and write our own
> files, but some we have in either netcdf3 or HDF4 files in TDS? Do
> aggregations work over different file types, as long as the variable name and
> the variable to aggregated over are the same?
>
> 3. In past incarnations, the chlorophyll files did not contain the two
> carbon variables POC, PIC, these were in separate files. Now they are
> included in the chlorophyll file. Is there any way that anyone can think of
> to continue that PIC and POC aggregations with the new files (remember I
> have use the same files for the chlorophyll aggregations) or I am pretty much
> stuck having to redownload the re-processed files?
>
> Thanks,
>
> -Roy
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>
> _______________________________________________
> thredds mailing list
> thredds@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
**********************
"The contents of this message do not reflect any position of the U.S.
Government or NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
