[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDFJava #MQO-415619]: Efficiently serializing NetCDF in memory objects

I am stopping further development on this until after it gets
into the master branch.
Your additional ideas seem useful, but do not seem essential.
I will revisit in the future.

> 
> Two more remarks/questions,
> 
> Currently, for files created or opened in "Diskless" mode, it is not possible 
> to get buffer by closing it with close_mem().
> I think capability could be interesting. What do you think?
> 
> Currently, for files created in "Inmemory" or "Diskless" mode, it is not 
> possible to create disk file by closing it with nc_close function.
> Same behavior for open inmemory  in writing mode.
> I think capability could be interesting. What do you think?
> 
> Regards
> 
> LANDIECH Matthieu
> 
> -----Message d'origine-----
> De : Unidata netCDF Java Support <address@hidden>
> Envoyé : vendredi 4 mai 2018 05:04
> À : address@hidden
> Cc : address@hidden; address@hidden; address@hidden; address@hidden; 
> address@hidden
> Objet : [netCDFJava #MQO-415619]: Efficiently serializing NetCDF in memory 
> objects
> 
> I just added a fix (I hope) for this problem.
> 
> >
> > I have launched your tests on in memory capability and the result of 
> > "nc_create_memio()" for netcdf-4 file seem wrong.
> > The return memory is "nil". I had the same behavior with the wrapped 
> > function.
> >
> > Bellow the obtained test report:
> >
> > *** Testing the inmemory API: netcdf-4.
> > ***Test open 1: nc_open_mem(): read-only ***Test open 2:
> > nc_open_memio(): read-only
> > finaldata: size=11537 memory=0x1fc05a0 ***Test open 3:
> > nc_open_memio(): read-write, copy
> > finaldata: size=11537 memory=0x1fe8560 ***Test open 4:
> > nc_open_memio(): read-write, locked, extra space
> > finaldata: size=262144 memory=0x1fe8560 ***Test create 1:
> > nc_create_memio(): no initialsize
> > finaldata: size=0 memory=(nil)
> > ***Test create 2: nc_create_memio(): initialsize; save file
> > finaldata: size=0 memory=(nil)
> > ***Test misc 1: use nc_close on created inmemory file ***Test misc 2:
> > use nc_close on opened inmemory file ***Test xfail 1: nc_open_mem():
> > write to read-only
> > ***XFail: status=-37 NetCDF: Write to read only ***Test xfail 2:
> > nc_open_memio(): modify without overallocating Suppressed because of
> > HDF5 library bug
> >
> > For netcdf-3, it seems to work.
> >
> > Regards,
> >
> > LANDIECH Matthieu
> >
> >
> > -----Message d'origine-----
> > De : Unidata netCDF Java Support
> > <address@hidden>
> > Envoyé : mercredi 25 avril 2018 23:26
> > À : address@hidden
> > Cc : address@hidden; address@hidden;
> > address@hidden; address@hidden;
> > address@hidden Objet : [netCDFJava #MQO-415619]: Efficiently
> > serializing NetCDF in memory objects
> >
> > You are correct. There are two versions here:
> > 1. the file format: netcdf-3 or netcdf-4 2. the software version. The 
> > current latest release is 4.6.1.
> > Do not be confused by the similarity of 4.5.2 to netcdf-4.
> > Regarding compatibility, we have also guaranteed that nay new software 
> > release can read and write both netcdf-3 and netcdf-4 as currently defined.
> > We will not change the formats any time soon (if ever).
> >
> > The second version is, as you note, just an increasing sequence of version 
> > numbers. As a rule, a higher version contains bug fixes and new features 
> > (such as inmemory) not found in lower numbered versions. so it is not 
> > exactly semantic versioning.
> >
> > The issue is how fast
> > you need the new inmemory feature. If you can wait some period of time, 
> > version to be named either 4.6.2 or 4.7.0, but in nay case, the one that 
> > will follow 4.6.1, then that new release should have the new memory 
> > features in it.
> >
> > I wish I could give a more exact release date for the next version, but we 
> > are running a little slow, and it takes time to go thru the set of bug 
> > fixes and new features before they can be incorporated into the master 
> > branch, which eventually becomes the new release.
> >
> > At some point the new memory stuff will have been merged into the master 
> > branch and you can treat that branch as equivalent to a beta release of th 
> > next release. I will certainly notify you when the memory features get 
> > incorporated into the master branch.
> >
> >
> >
> > >
> > > Thank you for your quick answer.
> > >
> > > I am not too familiar with your versioning scheme. Is it semantic
> > > versioning ?
> > >
> > > Here is my current understanding. I think NetCDF version groups
> > > different distinct items:
> > >
> > > - The NetCDF file format specification (which has not changed
> > > between releases 3.x and 4.y for example)
> > > - The NetCDF supporting software libraries
> > >
> > > As a user, I cannot lose my legacy data, which is in the NetCDF4
> > > file format for the foreseeable future. I am required to produce
> > > NetCDF4 data from NetCDF4 data. So, in theory, I can use any NetCDF
> > > software release as long as it operates on the NetCDF4 file format.
> > > It is not clear to me, but maybe the 5.x releases will be fully 
> > > compatible that format.
> > >
> > > Still, a major version upgrades will be a pain to justify. People
> > > will argue that:
> > >
> > > 1. It is still beta software;
> > > 2. the requested feature alone does not justify a major version bump
> > > according to semantic versioning; 3. long term guarantees might be
> > > required regarding NetCDF4 compatibility.
> > >
> > > So, in a nutshell, I would prefer this feature to be available in an
> > > upcoming NetCDF Java 4.x release.
> > >
> > > Kind regards,
> > >
> > > Michaël
> > >
> > > address@hidden>:
> > >
> > > > Yes, I got them; have not yet had a chance to review them.
> > > > I am still waiting to get the changes integrated into the netcdf-c
> > > > master branch. I cannot say when the next release will be, but I
> > > > would guess in about a month. In the meantime, I can get the API
> > > > changes into the netcdf-Java code. It will be ok as long as it is
> > > > only invoked with a version of the netcdf-c library that supports
> > > > the new functionality.
> > > > But I need to check again. are you still using netcdf-Java 4.6, or
> > > > are using the 5.0 beta?
> > > >
> > > > >
> > > > > My colleague Matthieu Landiech has sent you an API proposal and
> > > > > several questions regarding this feature.
> > > > > Have you received them ?
> > > > >
> > > > > What is the open window to get this feature in the next NetCDF
> > > > > Java release, scheduled in may ?
> > > > > Can you confirm when the modifications you made in the C library
> > > > > will be released ?
> > > > >
> > > > > Regards,
> > > > >
> > > > > Michaël
> > > > >
> > > > >
> > > > > address@hidden>:
> > > > >
> > > > > > You should work from the 5.0.0 branch.
> > > > > > I think the next/first release of 5.0.0 is scheduled for May.
> > > > > > =Dennis Heimbigner
> > > > > >   Unidata
> > > > > >
> > > > > >
> > > > > > Ticket Details
> > > > > > ===================
> > > > > > Ticket ID: MQO-415619
> > > > > > Department: Support netCDF Java
> > > > > > Priority: Critical
> > > > > > Status: Open
> > > > > > ===================
> > > > > > NOTE: All email exchanges with Unidata User Support are
> > > > > > recorded in the Unidata inquiry tracking system and then made
> > > > > > publicly available
> > > > through
> > > > > > the web.  If you do not want to have your interactions made
> > > > > > available
> > > > in
> > > > > > this way, you must let us know in each email you send to us.
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > >
> > > > =Dennis Heimbigner
> > > >   Unidata
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: MQO-415619
> > > > Department: Support netCDF Java
> > > > Priority: Critical
> > > > Status: Open
> > > > ===================
> > > > NOTE: All email exchanges with Unidata User Support are recorded
> > > > in the Unidata inquiry tracking system and then made publicly
> > > > available through the web.  If you do not want to have your
> > > > interactions made available in this way, you must let us know in each 
> > > > email you send to us.
> > > >
> > > >
> > > >
> > >
> > >
> >
> > =Dennis Heimbigner
> > Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: MQO-415619
> > Department: Support netCDF Java
> > Priority: Critical
> > Status: Closed
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> >
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: MQO-415619
> Department: Support netCDF Java
> Priority: Critical
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: MQO-415619
Department: Support netCDF Java
Priority: Critical
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.