The SMIs look ok. In principle you can aggregate over file types is they
are truly homogenous (same variable names, dimensions, coordinate systems,
etc). You really have to look hard at the result and test. You might be
better off starting a new aggregation where the new files start.
>From CDM POV, groups are ok, but they complicate the coordinate system
discovery, unless you are careful. In practice, the dataset writers are
often not careful.
On Mon, Jun 1, 2015 at 8:36 PM, Roy Mendelssohn - NOAA Federal <
roy.mendelssohn@xxxxxxxx> wrote:
> Hi John:
>
> Thanks.
>
> The L2 files we process and make our own composite files so they aren’t an
> issue It is mainly the L3’s (SMI) that we are concerned about. As I
> said, the two issues are just can they be served by TDS, forget any other
> problems (and we can do what we have been doing is add time as a coordinate
> in an NcML file which is what we then aggregate).
>
> But the bigger issue is whether we can aggregate over the different file
> types. If we are actually aggregating NcML files (which is usually the
> case), will TDS care that some of the files being pointed to in the NcML
> are HDF4 and some are netcdf4? If TDS doesn’t care that the filetypes are
> different, we probably can kludge a solution to a lot of the issues I
> raised. Otherwise, I may have to figure out how to re-download a ton of
> files over a bandwidth that is already often over-saturated.
>
> And can I open up the can of worms again? I am curious what people feel
> about the use of groups in netcdf4 files? It is no longer a theoretical
> question. If you are getting ocean color data from NASA, that is what you
> are getting. If people are interested I can post the file headers (or
> maybe more appropriately move this discussion to the netcdf mail-list) so
> you can see how they are being used. If you are familiar with the old HDF4
> format, it is mostly a mapping of that structure.
>
> -Roy
>
>
> > On Jun 1, 2015, at 7:20 PM, John Caron <caron@xxxxxxxx> wrote:
> >
> > Hi Roy:
> >
> > The grids look ok (L3m), although they didnt put the time values into a
> coordinate, but in global attributes.
> >
> > The swaths (L2) can probably be fixed up with a little NcML. Its not
> clear if aggregation is possible or useful.
> >
> > The points (L3b) are incomprehensible to me. Someone expert would have
> to explain whats in there.
> >
> > Not a lot of CF compliance that I can see.
> >
> > John
> >
> > On Mon, Jun 1, 2015 at 5:24 PM, Roy Mendelssohn - NOAA Federal <
> roy.mendelssohn@xxxxxxxx> wrote:
> > HI All:
> >
> > I don’t know how many of you get notices like this:
> >
> > > MODIS Ocean Color and SST Data Users:
> > >
> > > Within the next week, the Ocean Biology Processing Group will begin
> the ocean color (OC) reprocessing of MODIS on Aqua (MODISA). At that time,
> we will also make available the fully reprocessed SST from MODIS on Aqua
> and Terra, and we will start producing forward-stream products in the
> reprocessing configuration for all four data streams (MODIS SST and OC from
> both satellite platforms). General information on the reprocessing changes
> for OC and SST can be found here:
> >
> > The new format, if you don’t know, are netcdf-4 files, but they make
> heavy use of groups. As some of you may remember, a proposal to develop a
> standard for use of groups in CF was put forward by Charlie Zender and Ted
> Haberman, but got shot down. Be that as it may, the NASA product is what
> it is, and this raised some questions for our data services. To wit:
> >
> > 1. Does anyone know if TDS can handle these files correctly (because of
> the groups and not being strictly CF)? toolsUI seems to be able to read
> them okay, but I am not certain that it is picking up the data types
> properly (so recognizes that something is an array but not necessarily a
> grid)? Sample files are at:
> >
> > http://oceancolor.gsfc.nasa.gov/DOCS/FormatChange.html
> >
> >
> > 2. For some of the data we do some further processing and write our own
> files, but some we have in either netcdf3 or HDF4 files in TDS? Do
> aggregations work over different file types, as long as the variable name
> and the variable to aggregated over are the same?
> >
> > 3. In past incarnations, the chlorophyll files did not contain the two
> carbon variables POC, PIC, these were in separate files. Now they are
> included in the chlorophyll file. Is there any way that anyone can think
> of to continue that PIC and POC aggregations with the new files (remember
> I have use the same files for the chlorophyll aggregations) or I am pretty
> much stuck having to redownload the re-processed files?
> >
> > Thanks,
> >
> > -Roy
> >
> > **********************
> > "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> > **********************
> > Roy Mendelssohn
> > Supervisory Operations Research Analyst
> > NOAA/NMFS
> > Environmental Research Division
> > Southwest Fisheries Science Center
> > ***Note new address and phone***
> > 110 Shaffer Road
> > Santa Cruz, CA 95060
> > Phone: (831)-420-3666
> > Fax: (831) 420-3980
> > e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
> >
> > "Old age and treachery will overcome youth and skill."
> > "From those who have been given much, much will be expected"
> > "the arc of the moral universe is long, but it bends toward justice"
> -MLK Jr.
> >
> > _______________________________________________
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
>
> **********************
> "The contents of this message do not reflect any position of the U.S.
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new address and phone***
> 110 Shaffer Road
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK
> Jr.
>
>
