[netcdf-java] Opening a GridDataset for large NcML aggregations

[John Caron]

that should all work. the time may be in the scanning of 1000 files. Still, I have had other reports 
of extra slowness,  I will investigate when I can.

Jon Blower wrote:
> Hi John,
> 
> It's a JoinNew, because the data files do not contain a time axis.
> Hence we use a <scan> element and the time axis values are deduced
> from the file names using dateFormatMark.  Our ncml file is attached.
> 
> Yes, this is in our own application (ncWMS) not the TDS.
> 
> Cheers, Jon
> 
> On Tue, Mar 18, 2008 at 2:21 AM, John Caron <caron at unidata.ucar.edu> wrote:
>> HiJon:
>>
>>  Are these JoinExisting or JoinNew ? How is the coordinate value specified? You are doing this in your own application, not in TDS?
>>
>>
>>  Jon Blower wrote:
>>  > Hi all,
>>  >
>>  > Using the Java NetCDF libraries, version 2.2.22, we have notice that
>>  > opening a GridDataset using TypedDatasetFactory.open() is slow when
>>  > the dataset in question is an NcML aggregation over a long timeseries
>>  > (around a thousand timesteps, with one timestep per file).  The call
>>  > to TDF.open() takes around 3 seconds every time, which is a problem
>>  > for our application, where performance is important.
>>  >
>>  > We are using the NetcdfDatasetCache, which doesn't seem to help with
>>  > this problem.  Is there something we can do to speed up the process of
>>  > creating a GridDataset?  I don't think this was a problem for us in
>>  > previous library versions, where we used a different syntax
>>  > ("GridDataset gd = new GridDataset(nc);").
>>  >
>>  > I guess we could cache the GridDataset object in memory at the
>>  > application level, but is there a reason not to do this (e.g. if this
>>  > is a very large object)?
>>  >
>>  > Here's what we do using version 2.2.22:
>>  >
>>  >    NetcdfDataset nc = NetcdfDataset nc = NetcdfDatasetCache.acquire(filename);
>>  >    GridDataset gd =
>>  > (GridDataset)TypedDatasetFactory.open(DataType.GRID, nc, null, null);
>>  >
>>  > Thanks, Jon
>>  >
>>  _______________________________________________
>>  netcdf-java mailing list
>>  netcdf-java at unidata.ucar.edu
>>  For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/
>>
> 
> 
>