Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.

Re: [netcdf-java] Java API: problems with temporary files

Bill-

Bill Moninger wrote:
I just today dealt with this problem, but for me the potential race condition remains. 

You can specify where the temporary files are placed with
DiskCache.setRootDirectory("grib_data");


In the IDV, we use this feature to keep the index files out the
working directory.


You can purge that directory with
DiskCache.cleanCache(new java.util.Date(),null);
That does an immediate purge, but you could specify a later date that might minimize the potential race condition. 

We purge temp files (uncompressed files, gbx) after 24 hours.  The
IDV runs through the files in the cache and deletes files older than
that.
I've only just tried this for a few hours. It works for keeping the *.gbx files out of our data directories. 

There's probably a better solution, and I'm looking forward to seeing it.


I don't know of a better solution, but here are a couple of other points
on grib indexing:

- If there is a .gbx file located in the same directory as the file
itself, the library will use that and ignore the temp file stuff.
I use this for a test GFS Global grid so I don't have to re-index
when my temp space gets cleared out.  It takes several minutes to
index this file so once the index was created in my temp space, I
copied it over to the same directory as the grib file.

- If you have a grib file, say test.grb and index it,then rewrite
test.grb but it ends up being the same size (maybe you are just changing
param tables), the index will not get rewritten automatically.  There is
a method that can be used to set this to always rewrite the index if the
date has changed, but I can't find docs on what it is.  Perhaps
it's GribServiceProvider.setIndexSyncMode()?  It would be nice to have
some javadocs filled in on all these methods, so we know what they
do.  John?

- As John mentioned, an index is valid for a particular file. I would
be very leary of using the same index for different files because it
holds the offsets to each grib message and sections of each grib message
(PDS, GDS, etc) in the file and info about the parameter, projection, time, etc. The files would have to be identical, not just similar. 

Don Murray

On 2/9/2009 12:41 PM, Valliappa Lakshmanan wrote:
I've been using the Java Netcdf API and am running into several issues with the temporary files that are created. 

For example, when a gzipped NetCDF file is read, an uncompressed
version is written to disk before it is read. I assume that this is because the NetCDF API allows for file seeks etc. which wouldn't be possible if the file remained gzipped. But (and this is the problem), the uncompressed file is stored in the SAME directory as the original file. This causes three major problems (in increasing order 
of severity):
(a) The data directory (which would be considered read-only) is being modified. This causes problems 
        because of the I/O optimizations that we do on real-time systems
(b) The temporary file is not automatically cleaned up, so these temporaries start to fill up the disk. But removing the temporary file when the original file is closed is not enough because of problem (c). (c) if there are two simultaneous programs reading a gzipped file, then a potential for race conditions exists
The same problem seems to exist when reading a Grib1 file (a .gbx temporary file is created). 

Is there any work-around for this problem?

Lak

p.s.  I suggest the consistent use of java.io.File's createTempFile() ...


------------------------------------------------------------------------

_______________________________________________
netcdf-java mailing list
netcdf-java@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ 



--
*************************************************************
Don Murray                               UCAR Unidata Program
dmurray@xxxxxxxxxxxxxxxx                        P.O. Box 3000
(303) 497-8628                              Boulder, CO 80307
http://www.unidata.ucar.edu/staff/donm
*************************************************************
  • 2009 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: