[netcdf-java] Persisting a NetcdfDatasetCache

[Nick Bower]

John Caron wrote:
> Hi Jon and Nick:
>
> Ive been considering whether an object cache like ehcache would be the way to go, but havent had
> time to investigate.
>
> 1. Since NetcdfFileCache/NetcdfDatasetCache encapsolates an open file handle, I need to control the
> number of such handles. Ive assumed that i want to avoid the overhead of opening and closing files,
> but havent actually timed this (BAD programmer!).
>
> 2. The first few versions of netcdf java were serializable, but ive stopped supporting that for
> various reasons (eg Bloch "Effective Java" item 54). Could be revisted, but serialization is a pain.
>
> "Do not implement Serializable lightly, since it restricts future flexibility, and publicly exposes
> class implementation details which are usually private. As well, implementing Serializable correctly
> is not trivial."
>
> "Serialization demands not only backward compatibility, but forward compatibility. You might have
> people running slightly older versions of software needing to read the newer versions, and vice
> versa. Serialization incompatibilities are even more serious than API incompatibilities: we are
> talking about customers not being able to read data that they innocently stored away with a previous
> version of your software!"
>   
Random thoughts -

I think those comments above are wise in urging caution for people 
expecting to use serialization 1) long-term persistence with something 
like built-in ObjectOutputStream, 2) serialization for any form of 
exchange between software components.

But regarding the concerns with migration issues, we're not talking 
about long-term persistence here or data exchange (eg messaging) - more 
a very short-term store that is highly disposable and likely to be 
discarded and rebuilt upon any migration.  And my understanding is that, 
given the need to read/write netcdf files across a variety of platforms, 
the point is that the format does not change within a version - v3 is v3 
as specified (I'm guessing).

Perhaps the IOSPs provide some method for serialziation already?  It 
doesn't have to be just using EHCache/ObjectOutputStream directly on 
NetcdfDataset of course.  You could be caching a more abstract notion of 
a netcdf file that lazy instantiates and re-opens file handle as 
needed.  May require some refactoring though as having NetcdfDataset 
directly extend NetcdfFile could be an issue.

Sorry I can't be of more help.  To be honest I don't have a great 
understanding of the code internals.