[netcdf-java] Opening a GridDataset for large NcML aggregations

[John Caron]

Aggregations are done when NetcdfDataset is opened, which should get cached. But I remember noticing 
that the aggregation was not getting cached. I will check it out.

Jon Blower wrote:
> Hi John,
> 
> Yes, I thought it might be the scanning that takes the time.  I had
> assumed that the scanning takes place when you open a NetcdfDataset,
> but I guess I'm wrong?  Perhaps this implies that I need a
> GridDatatypeCache as well as a NetcdfDatasetCache?  (Caching is making
> my head hurt, there are too many in my application already ;-)
> 
> Cheers, Jon
> 
> On Tue, Mar 18, 2008 at 4:22 PM, John Caron <caron at unidata.ucar.edu> wrote:
>> that should all work. the time may be in the scanning of 1000 files. Still, I have had other reports
>>  of extra slowness,  I will investigate when I can.
>>
>>
>>
>>  Jon Blower wrote:
>>  > Hi John,
>>  >
>>  > It's a JoinNew, because the data files do not contain a time axis.
>>  > Hence we use a <scan> element and the time axis values are deduced
>>  > from the file names using dateFormatMark.  Our ncml file is attached.
>>  >
>>  > Yes, this is in our own application (ncWMS) not the TDS.
>>  >
>>  > Cheers, Jon
>>  >
>>  > On Tue, Mar 18, 2008 at 2:21 AM, John Caron <caron at unidata.ucar.edu> wrote:
>>  >> HiJon:
>>  >>
>>  >>  Are these JoinExisting or JoinNew ? How is the coordinate value specified? You are doing this in your own application, not in TDS?
>>  >>
>>  >>
>>  >>  Jon Blower wrote:
>>  >>  > Hi all,
>>  >>  >
>>  >>  > Using the Java NetCDF libraries, version 2.2.22, we have notice that
>>  >>  > opening a GridDataset using TypedDatasetFactory.open() is slow when
>>  >>  > the dataset in question is an NcML aggregation over a long timeseries
>>  >>  > (around a thousand timesteps, with one timestep per file).  The call
>>  >>  > to TDF.open() takes around 3 seconds every time, which is a problem
>>  >>  > for our application, where performance is important.
>>  >>  >
>>  >>  > We are using the NetcdfDatasetCache, which doesn't seem to help with
>>  >>  > this problem.  Is there something we can do to speed up the process of
>>  >>  > creating a GridDataset?  I don't think this was a problem for us in
>>  >>  > previous library versions, where we used a different syntax
>>  >>  > ("GridDataset gd = new GridDataset(nc);").
>>  >>  >
>>  >>  > I guess we could cache the GridDataset object in memory at the
>>  >>  > application level, but is there a reason not to do this (e.g. if this
>>  >>  > is a very large object)?
>>  >>  >
>>  >>  > Here's what we do using version 2.2.22:
>>  >>  >
>>  >>  >    NetcdfDataset nc = NetcdfDataset nc = NetcdfDatasetCache.acquire(filename);
>>  >>  >    GridDataset gd =
>>  >>  > (GridDataset)TypedDatasetFactory.open(DataType.GRID, nc, null, null);
>>  >>  >
>>  >>  > Thanks, Jon
>>  >>  >
>>  >>  _______________________________________________
>>  >>  netcdf-java mailing list
>>  >>  netcdf-java at unidata.ucar.edu
>>  >>  For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
>>  >>
>>  >
>>  >
>>  >
>>
> 
> 
>