Re: [netcdf-java] fastest way to determine feature type

Hi Don,

Do your files happen to have an unlimited dimension when it is not required?

In the past we've had a performance issues dealing with static data sets only 
to later realize the slow load times where due to the reading of data 
associated with an unlimited dimension (i.e. "time").  When a dimension is 
unlimited the values associated with it are stored sparsely though out the the 
file. Converting the unlimited dimension to fixed significantly increased 
time-to-open these files (i.e. the values for the "time" axis are stored 
contiguously, no longer sparsely).  You'll want unlimited if you intend to 
append data along that dimension to the file in the future, otherwise make sure 
it's fixed if you are concerned about performance on initial open.

Tom Kunicki
Center for Integrated Data Analytics
U.S. Geological Survey
8505 Research Way
Middleton, WI  53562

On Jun 21, 2012, at 4:13 PM, Don Murray wrote:

> Just as a followup, the attached program tests the speed of opening a file 
> using the method in FeatureScan vs.  In my test, the latter 
> is actually faster by a few milliseconds.  The real slowdown is the initial 
> os caching of the file (in this case a 3.3 GB file). Once the file is in the 
> OS cache, both methods are pretty quick.
> Thanks to John (and Roland) for their help.
> Don
> On 6/20/12 8:14 PM, John Caron wrote:
>> On 6/19/2012 3:19 PM, Don Murray wrote:
>>> Hi-
>>> I have a bunch of netCDF files and I want to quickly determine whether
>>> they are grids, trajectories, or point features.  For grids, I've been
>>> using GridDataset gds = and catch the exception
>>> if it's not a grid, but for a 3.3 GB file, that can take 2 minutes (or
>>> longer) to open and create the dataset if it is a grid.  I was
>>> wondering if there's a quicker method of determining the feature type
>>> of a netCDF file.
>>> Thanks for your help.
>>> Don
>> Hi Don:
>> The most convenient thing is to use ToolsUI / FeatureTypes /
>> FeatureScan, and give it a file or directory. It will try to figure out
>> the type and report on what it finds.
>> The code is in, you can copy the parts
>> you need.
>> Its an ongoing process, i think im not doing it as well as it can be
>> done. Send me reports on files it misidentifies.
>> John
>> _______________________________________________
>> netcdf-java mailing list
>> netcdf-java@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
> -- 
> Don Murray
> 303-497-3596
> <>_______________________________________________
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit: 

  • 2012 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: