Re: [netcdf-java] Persisting a NetcdfDatasetCache

Hi Nick and John (and list),

I was about to write pretty much exactly what Nick just wrote - the
serialization would be for short-term storage so I don't think there's
a problem there (although I guess some people might try to misuse the
serialization capability if it were provided).

My _guess_ is that there is no need to keep open file handles and the
overhead of opening a NetcdfDataset is mostly in the reading of the
metadata into the relevant structures (Attributes, Axes etc),
particularly if the dataset is an aggregation (actually it was
aggregations that I had in mind when I suggested the persistent store
in the first place).  However, some testing would be needed of course
- I don't really know.

Maybe the metadata could be stored in some other way than
serialization. We have done some analogous experiments with trying to
use a relational database to store metadata but the problem is that
the database gets complicated when you start dealing with projections
and the like.  Object-relational mapping could help to manage the
complexity but this won't be trivial to code (e.g. could use Hibernate
over an embedded database like H2).

Cheers, Jon

On Thu, Feb 21, 2008 at 12:36 AM, Nick Bower
<nick.bower@xxxxxxxxxxxxxxxxxxxxx> wrote:
> John Caron wrote:
>  > Hi Jon and Nick:
>  >
>  > Ive been considering whether an object cache like ehcache would be the way 
> to go, but havent had
>  > time to investigate.
>  >
>  > 1. Since NetcdfFileCache/NetcdfDatasetCache encapsolates an open file 
> handle, I need to control the
>  > number of such handles. Ive assumed that i want to avoid the overhead of 
> opening and closing files,
>  > but havent actually timed this (BAD programmer!).
>  >
>  > 2. The first few versions of netcdf java were serializable, but ive 
> stopped supporting that for
>  > various reasons (eg Bloch "Effective Java" item 54). Could be revisted, 
> but serialization is a pain.
>  >
>  > "Do not implement Serializable lightly, since it restricts future 
> flexibility, and publicly exposes
>  > class implementation details which are usually private. As well, 
> implementing Serializable correctly
>  > is not trivial."
>  >
>  > "Serialization demands not only backward compatibility, but forward 
> compatibility. You might have
>  > people running slightly older versions of software needing to read the 
> newer versions, and vice
>  > versa. Serialization incompatibilities are even more serious than API 
> incompatibilities: we are
>  > talking about customers not being able to read data that they innocently 
> stored away with a previous
>  > version of your software!"
>  >
>  Random thoughts -
>
>  I think those comments above are wise in urging caution for people
>  expecting to use serialization 1) long-term persistence with something
>  like built-in ObjectOutputStream, 2) serialization for any form of
>  exchange between software components.
>
>  But regarding the concerns with migration issues, we're not talking
>  about long-term persistence here or data exchange (eg messaging) - more
>  a very short-term store that is highly disposable and likely to be
>  discarded and rebuilt upon any migration.  And my understanding is that,
>  given the need to read/write netcdf files across a variety of platforms,
>  the point is that the format does not change within a version - v3 is v3
>  as specified (I'm guessing).
>
>  Perhaps the IOSPs provide some method for serialziation already?  It
>  doesn't have to be just using EHCache/ObjectOutputStream directly on
>  NetcdfDataset of course.  You could be caching a more abstract notion of
>  a netcdf file that lazy instantiates and re-opens file handle as
>  needed.  May require some refactoring though as having NetcdfDataset
>  directly extend NetcdfFile could be an issue.
>
>  Sorry I can't be of more help.  To be honest I don't have a great
>  understanding of the code internals.
>
>
>
>
>  _______________________________________________
>  netcdf-java mailing list
>  netcdf-java@xxxxxxxxxxxxxxxx
>  For list information or to unsubscribe, visit: 
> http://www.unidata.ucar.edu/mailing_lists/
>



-- 
--------------------------------------------------------------
Dr Jon Blower Tel: +44 118 378 5213 (direct line)
Technical Director Tel: +44 118 378 8741 (ESSC)
Reading e-Science Centre Fax: +44 118 378 6413
ESSC Email: jdb@xxxxxxxxxxxxxxxxxxxx
University of Reading
3 Earley Gate
Reading RG6 6AL, UK
--------------------------------------------------------------


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: