Oops, I switched answers. I have a working, but not completely tested version of this. Attached is the current documentation for it. > > Thank you for making progress on this feature. I am not sure I understand > your current difficulty correctly. > > Our goal is to allow simple serialization of NetCDF objects over > communication buses by directly accessing the underlying memory buffer . We > found the HDF5 library offers the H5FD_CORE driver for in memory read and > writes (see documentation > <https://support.hdfgroup.org/HDF5/doc/UG/HDF5_Users_Guide-Responsive%20HTML5/index.html#t=HDF5_Users_Guide%2FTheFile%2FThe_HDF5_File.htm%23TOC_3_11_Alternate_Filebc-17&rhtocid=3.3>). > So, we need to > > 1. Modify the NetCDF library to optionally use this driver > 2. Add a mechanism to the NetCDF library so the underlying memory > buffer, if it is not already possible > > Then, i remember we discussed several performance optimization > opportunities. > > Are you encountering issues using the H5FD_CORE driver from the NetCDF > library ? Or are you already working on the performance optimizations ? > > Do not hesitate to ask if I can perform some implementation or testing work > on your behalf. I am really interested by this work and want to contribute. > > Kind regards, > > Michaël > > address@hidden>: > > > I have been experimenting with this. I rebuilt the libsrc4 > > (netcdf4) metadata code and have been trying various things. > > I am worried that the cost of reading the metadata > > is completely driven by the hdf5 library and that almost nothing > > I can do at the netcdf library level will have much effect. I hope > > I am wrong. The test I need is to have an hdf5 program > > that just times the cost to read (but not store) all of the > > metadata from an hdf5 file. Then compare that time to that of > > a simple netcdf program that just opens the same file (which > > implicitly reads all of the metadata). > > I have a program to generate a file with a large amount of metadata > > (determined by parameters in the program) and am using that as > > the basis for my experiments. > > > > > > > > Any progress on this feature ? > > > > > > Michaël > > > > > > 2018-01-30 18:31 GMT+01:00 Michaël Melchiore <address@hidden>: > > > > > > > Thank you. > > > > I will need time to setup so I can test your changes. Please keep me > > > > posted :) > > > > > > > > Michaël > > > > > > > > address@hidden>: > > > > > > > >> Thanks for the reminder; this fix got lost > > > >> in the flurry of changes for our most recent release. > > > >> I will move it up in the priority queue. > > > >> > > > >> > > > > >> > Any news on the implementation progress ? > > > >> > Can I help ? > > > >> > > > > >> > Kind regards, > > > >> > > > > >> > Michaël > > > >> > > > > >> > 2018-01-03 10:56 GMT+01:00 Michaël Melchiore <address@hidden>: > > > >> > > > > >> > > Yes, of course. > > > >> > > > > > >> > > I will need to setup for benchmarking, so if you can give me an > > > >> estimated > > > >> > > target date/week, this would help me organize. > > > >> > > > > > >> > > Michaël > > > >> > > > > > >> > > address@hidden>: > > > >> > > > > > >> > >> I will go ahead and implement as I described. At that point > > > >> > >> I will need you to help me see if it meets your performance > > needs. > > > >> > >> > > > >> > >> > Thank you for the clarification. > > > >> > >> > > > > >> > >> > So, writing to a in-memory files requires at least copying > > > >> incoming data > > > >> > >> > into the memory buffer. I think this is perfectly OK as long as > > > >> HDF5 > > > >> > >> does > > > >> > >> > not copy or reallocate too much extra data. > > > >> > >> > > > > >> > >> > If input data does not fit in the buffer, someone will > > obviously > > > >> have to > > > >> > >> > move memory around. We just need to manage size hints and let > > users > > > >> > >> provide > > > >> > >> > sensible values. > > > >> > >> > My comment was more about reducing internal copies introduced > > by > > > >> NetCDF > > > >> > >> and > > > >> > >> > /or HDF5 implementations since users would have no control over > > > >> them. > > > >> > >> > > > > >> > >> > HDF5 is designed for efficient I/O and for high volume and > > complex > > > >> > >> data. So > > > >> > >> > I think we should not worry too much here. > > > >> > >> > > > > >> > >> > Regards, > > > >> > >> > > > > >> > >> > Michaël > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > > >> > >> > address@hidden>: > > > >> > >> > > > > >> > >> > > The rule I am currently proposing is that > > > >> > >> > > if the memory file is opened read-only, then no copy > > > >> > >> > > is made and ownership remains with the caller. > > > >> > >> > > If it is writable then a copy is made internally and owned > > by the > > > >> > >> netcdf > > > >> > >> > > library. > > > >> > >> > > The reason for doing this is to reduce memory. It is also > > > >> consistent > > > >> > >> > > with the way HDF5 works. > > > >> > >> > > This is admittedly a somewhat complicated rule, so the > > > >> alternative > > > >> > >> > > is for the netcdf-c library to always make an internal copy > > of > > > >> the > > > >> > >> incoming > > > >> > >> > > data. In any case, the memory returned by nc_close_mem() > > would > > > >> need > > > >> > >> to be > > > >> > >> > > free'd by the caller. > > > >> > >> > > > > > >> > >> > > > My understanding is that 1) would incur no copy. For 2), it > > > >> seems at > > > >> > >> > > least > > > >> > >> > > > one mandatory copy would me made. Am I correct ? > > > >> > >> > > Any time you write to an in-memory file, it will incur at > > least > > > >> one > > > >> > >> copy > > > >> > >> > > internally. > > > >> > >> > > > > > >> > >> > > >I t also seems nc_create is missing an argument to provide a > > > >> size > > > >> > >> hint for > > > >> > >> > > > the memory buffer. This would help avoiding unnecessary > > > >> > >> reallocations. > > > >> > >> > > Good idea. > > > >> > >> > > > > > >> > >> > > > Unnecessary copies will kill any potential improvements > > > >> > >> > > > from the new API. > > > >> > >> > > Possibly true, but I do not think I have enough > > > >> > >> > > control over the HDF5 implementation to do this when > > > >> > >> > > the in-memory file is modified. No-copy is only possible > > > >> > >> > > if the file is read-only. > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > > >> > >> > > > Does nc_open_mem takes ownership of the memory ? In other > > > >> words, > > > >> > >> who is > > > >> > >> > > > responsible for releasing the provided memory ? (It seems > > it > > > >> is the > > > >> > >> > > caller > > > >> > >> > > > since you mention a copy is made internally) > > > >> > >> > > > > > > >> > >> > > > 2. The memory returned by nc_close_mem would be a COPY of > > the > > > >> > >> current > > > >> > >> > > > > contents if the original was writeable (again HDF5 > > > >> forced). > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > Is it a copy of initial data (provided through nc_open_mem > > for > > > >> > >> example) > > > >> > >> > > or > > > >> > >> > > > is it a copy of the internal copy ? Also, can you clairify > > > >> what the > > > >> > >> > > > original is in that context ? > > > >> > >> > > > > > > >> > >> > > > I have two main use cases for this feature > > > >> > >> > > > > > > >> > >> > > > 1. Deserialization: open a read-only NetCDF from memory and > > > >> provide > > > >> > >> it > > > >> > >> > > > as read-only input for algorithms > > > >> > >> > > > 2. Serialization: NetCDF is created and populated > > in-memory by > > > >> > >> > > > algorithms. Once data is submitted from algorithms to the > > > >> > >> framework, it > > > >> > >> > > is > > > >> > >> > > > treated as read-only. Framework would access memory to > > > >> efficiently > > > >> > >> > > > serialize it for transport > > > >> > >> > > > > > > >> > >> > > > My understanding is that 1) would incur no copy. For 2), it > > > >> seems at > > > >> > >> > > least > > > >> > >> > > > one mandatory copy would me made. Am I correct ? > > > >> > >> > > > > > > >> > >> > > > It also seems nc_create is missing an argument to provide a > > > >> size > > > >> > >> hint for > > > >> > >> > > > the memory buffer. This would help avoiding unnecessary > > > >> > >> reallocations. > > > >> > >> > > > > > > >> > >> > > > Regarding your thoughts, my use-case is about maximizing > > > >> > >> performance for > > > >> > >> > > > large datasets. Unnecessary copies will kill any potential > > > >> > >> improvements > > > >> > >> > > > from the new API. > > > >> > >> > > > So I would request a "no-copy" API, even if it comes with > > its > > > >> set of > > > >> > >> > > > constraints. The nc_close_mem API seems a good way to > > enforce > > > >> some > > > >> > >> of > > > >> > >> > > them. > > > >> > >> > > > > > > >> > >> > > > No/zero copy API are usually for advanced users and should > > be > > > >> > >> opt-in. One > > > >> > >> > > > advice I can give is to provide dedicated API for zero copy > > > >> > >> > > > > > > >> > >> > > > 1. Default API could incur copy, which is safe for normal > > users > > > >> > >> > > > 2. Dedicated API will give you more freedom for the design > > of > > > >> the > > > >> > >> > > > zero-copy function signatures > > > >> > >> > > > > > > >> > >> > > > So to answer your question, I would provide two sets of > > API if > > > >> you > > > >> > >> are > > > >> > >> > > > concerned about complexity of zero copy optimizations for > > your > > > >> > >> users. > > > >> > >> > > > > > > >> > >> > > > Regards, > > > >> > >> > > > > > > >> > >> > > > Michaël > > > >> > >> > > > > > > >> > >> > > > address@hidden>: > > > >> > >> > > > > > > >> > >> > > > > Proposal: > > > >> > >> > > > > I am in the process of adding support for retrieving > > > >> > >> > > > > the final file image for an in-memory file. This would > > > >> > >> > > > > extend the existing nc_open_mem functionality. > > > >> > >> > > > > > > > >> > >> > > > > This involves several changes. > > > >> > >> > > > > 1. nc_open_mem() would now allow the NC_WRITE mode flag > > > >> > >> > > > > so a chunk of memory can be passed in and be modified > > > >> > >> > > > > 2. nc_create() would now allow the NC_INMEMORY flag to > > be set > > > >> > >> > > > > to cause the created file to be kept in memory. > > > >> > >> > > > > 3. the ability would be provided to obtain the in-memory > > > >> contents > > > >> > >> > > > > of a file at the point it is closed using a new > > function > > > >> > >> > > > > nc_close_mem(int ncid, size_t* sizep, void** > > memory) > > > >> > >> > > > > Note the following. > > > >> > >> > > > > 1. If nc_open_mem is called with NC_WRITE, then a copy > > of the > > > >> > >> incoming > > > >> > >> > > > > memory would be made internally (dictated by HDF5 > > > >> > >> functionality). > > > >> > >> > > > > If instead, the file is read-only, no copy would be > > made . > > > >> > >> > > > > 2. The memory returned by nc_close_mem would be a COPY > > of the > > > >> > >> current > > > >> > >> > > > > contents if the original was writeable (again HDF5 > > > >> forced). > > > >> > >> > > > > > > > >> > >> > > > > I also have a couple of thoughts. > > > >> > >> > > > > 1. Instead of nc_close_mem, I could separate out the > > memory > > > >> > >> extraction > > > >> > >> > > > > from the close operation. Hence one would call > > something > > > >> like > > > >> > >> > > > > nc_get_mem() and then nc_close() instead of nc_close > > only. > > > >> > >> > > > > I prefer the nc_close_mem solution, but can anyone > > > >> provide a > > > >> > >> > > > > use case where the separate functions is better? > > > >> > >> > > > > 2. I could always make it so that the incoming and > > extracted > > > >> > >> memory > > > >> > >> > > > > chunks were copies. This would mean that read-only > > opening > > > >> > >> > > > > a memory chunk would pay the copy cost when it did not > > > >> have to. > > > >> > >> > > > > But the always copy soln might be simpler for the > > user to > > > >> > >> understand > > > >> > >> > > > > and use. > > > >> > >> > > > > Note that this is all preliminary and is subject to > > change > > > >> if, > > > >> > >> > > > > for example, I cannot get some piece of functionality to > > > >> work. > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > Sorry, I posted the proposal to the netcdf news group > > and > > > >> > >> forgot to > > > >> > >> > > copy > > > >> > >> > > > > > to the gihub site. I will do that ASAP. > > > >> > >> > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > This is good news. I am eager to provide feedback on > > your > > > >> > >> proposal. > > > >> > >> > > > > > > I did not find any proposal in the linked issue. Did > > you > > > >> > >> forget to > > > >> > >> > > > > push :) ? > > > >> > >> > > > > > > > > > >> > >> > > > > > > Also, it has not been updated with our latest > > > >> discussions. Do > > > >> > >> you > > > >> > >> > > want > > > >> > >> > > > > me > > > >> > >> > > > > > > to fix that ? > > > >> > >> > > > > > > > > > >> > >> > > > > > > Regards, > > > >> > >> > > > > > > > > > >> > >> > > > > > > Michaël > > > >> > >> > > > > > > > > > >> > >> > > > > > > 2017-12-11 22:19 GMT+01:00 Unidata netCDF Java > > > >> > >> > > > > > > address@hidden>: > > > >> > >> > > > > > > > > > >> > >> > > > > > > > It appears I can do this. You might examine my > > proposed > > > >> > >> > > > > > > > API and tell me if it will serve your purposes. > > > >> > >> > > > > > > > See https://github.com/Unidata/netcdf-c/issues/708 > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > I will investigate and see if I can use this > > function > > > >> > >> > > > > > > > > to get the desired effect. > > > >> > >> > > > > > > > > Thanks for bringing it to my attention. > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > The HDFgroup forum has pointed me to the > > > >> > >> H5Pset_file_image > > > >> > >> > > > > > > > > > <https://support.hdfgroup.org/ > > > >> HDF5/doc/RM/RM_H5P.html# > > > >> > >> > > > > > > > Property-SetFileImage> > > > >> > >> > > > > > > > > > function. This is documented further here > > > >> > >> > > > > > > > > > <https://support.hdfgroup.org/ > > HDF5/doc/Advanced/ > > > >> > >> > > > > FileImageOperations/ > > > >> > >> > > > > > > > HDF5FileImageOperations.pdf> > > > >> > >> > > > > > > > > > and seems to be available from HDF5 1.8.9 > > upwards . > > > >> > >> > > > > > > > > > So, I should rejoice as the limitations no > > longer > > > >> comes > > > >> > >> from > > > >> > >> > > the > > > >> > >> > > > > HDF5 > > > >> > >> > > > > > > > > > software. Yet, I am a bit suspicious because > > this > > > >> > >> feature has > > > >> > >> > > > > been > > > >> > >> > > > > > > > > > available from mid-2012 and I fear you are not > > > >> using it > > > >> > >> for a > > > >> > >> > > > > good > > > >> > >> > > > > > > > reason. > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > What do you think ? I am willing to try > > > >> implementing it > > > >> > >> if > > > >> > >> > > > > someone can > > > >> > >> > > > > > > > > > mentor me. > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > Regards, > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > Michaël > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > 2017-12-04 19:52 GMT+01:00 Unidata netCDF Java > > > >> > >> > > > > > > > > > address@hidden>: > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > I think that as long as the mmap is set up > > with > > > >> > >> > > non-persist, > > > >> > >> > > > > then > > > >> > >> > > > > > > > > > > the only writes to disk will occur with > > paging. > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > Actually, my solution is less than optimal > > > >> since, > > > >> > >> using > > > >> > >> > > > > mmap, data > > > >> > >> > > > > > > > is > > > >> > >> > > > > > > > > > > still > > > >> > >> > > > > > > > > > > > written to disk eventually. > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > The goal is to avoid both writes to disk > > and > > > >> data > > > >> > >> copies. > > > >> > >> > > > > This is > > > >> > >> > > > > > > > > > > sometimes > > > >> > >> > > > > > > > > > > > called "zero copy" optimization and would > > work > > > >> > >> because > > > >> > >> > > in my > > > >> > >> > > > > > > > application, > > > >> > >> > > > > > > > > > > > we enforce that input data is read only. > > Output > > > >> > >> data is > > > >> > >> > > > > modified > > > >> > >> > > > > > > > > > > obviously > > > >> > >> > > > > > > > > > > > but can be seen as read only just before > > > >> > >> transmission. > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > I have started a thread on the HDF forum > > on the > > > >> > >> topic. I > > > >> > >> > > am > > > >> > >> > > > > also > > > >> > >> > > > > > > > looking > > > >> > >> > > > > > > > > > > > into Apache Common VFS Ram filesystem as a > > > >> fallback > > > >> > >> > > > > workaround. > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > Regards, > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > 2017-12-04 18:51 GMT+01:00 Unidata netCDF > > Java > > > >> > >> > > > > > > > > > > > address@hidden>: > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > Actually, we do use mmap already. The > > > >> problem is > > > >> > >> that > > > >> > >> > > the > > > >> > >> > > > > mapped > > > >> > >> > > > > > > > memory > > > >> > >> > > > > > > > > > > > > is hidden deep in, for example, the hdf5 > > > >> memory > > > >> > >> > > driver. So > > > >> > >> > > > > there > > > >> > >> > > > > > > > is no > > > >> > >> > > > > > > > > > > > > way I can see to get access to it. If we > > did > > > >> have > > > >> > >> > > access > > > >> > >> > > > > to it, > > > >> > >> > > > > > > > then > > > >> > >> > > > > > > > > > > > > of course, we could copy it out and give > > you > > > >> the > > > >> > >> > > > > functionality > > > >> > >> > > > > > > > you > > > >> > >> > > > > > > > > > > desire . > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > My ideal use case would allow users to > > > >> > >> create/modify > > > >> > >> > > > > NetCDF4 > > > >> > >> > > > > > > > datasets > > > >> > >> > > > > > > > > > > > > > in-memory. Once done, my software would > > > >> read the > > > >> > >> > > binary > > > >> > >> > > > > data > > > >> > >> > > > > > > > and > > > >> > >> > > > > > > > > > > send it > > > >> > >> > > > > > > > > > > > > > over the network. > > > >> > >> > > > > > > > > > > > > > One workaround I see is to use POSIX > > memory > > > >> > >> mapped > > > >> > >> > > file > > > >> > >> > > > > > > > > > > > > > <https://en.wikipedia.org/wiki > > > >> > >> /Memory-mapped_file> > > > >> > >> > > to > > > >> > >> > > > > trick > > > >> > >> > > > > > > > the > > > >> > >> > > > > > > > > > > library > > > >> > >> > > > > > > > > > > > > > into working in-memory. > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > I now understand this issue originates > > > >> from a > > > >> > >> > > limitation > > > >> > >> > > > > of > > > >> > >> > > > > > > > the HFD5 > > > >> > >> > > > > > > > > > > > > > library. I can raise this issue to the > > HDF > > > >> > >> group. > > > >> > >> > > > > Publications > > > >> > >> > > > > > > > such > > > >> > >> > > > > > > > > > > as > > > >> > >> > > > > > > > > > > > > this > > > >> > >> > > > > > > > > > > > > > one <https://support.hdfgroup.org/ > > > >> > >> > > > > pubs/papers/Big_HDF_FAQs.pdf> > > > >> > >> > > > > > > > (see > > > >> > >> > > > > > > > > > > > > also > > > >> > >> > > > > > > > > > > > > > this > > > >> > >> > > > > > > > > > > > > > <https://www.hdfgroup.org/ > > > >> > >> > > 2015/03/from-hdf5-datasets-to- > > > >> > >> > > > > > > > > > > > > apache-spark-rdds/>) > > > >> > >> > > > > > > > > > > > > > seems to indicate they would be > > interested > > > >> bythe > > > >> > >> > > feature. > > > >> > >> > > > > > > > Indeed, > > > >> > >> > > > > > > > > > > > > avoiding > > > >> > >> > > > > > > > > > > > > > unecessary transfers to disk is key to > > > >> > >> achieving good > > > >> > >> > > > > > > > performance in > > > >> > >> > > > > > > > > > > Big > > > >> > >> > > > > > > > > > > > > > Data systems (this is the whole point > > of > > > >> Apache > > > >> > >> Spark > > > >> > >> > > > > BTW). > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > Regards, > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > 2017-12-03 4:36 GMT+01:00 Unidata > > netCDF > > > >> Java > > > >> > >> > > > > > > > > > > > > > address@hidden>: > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > This question has come up before. > > > >> > >> > > > > > > > > > > > > > > This is currently not possible for > > > >> netcdf-4 > > > >> > >> files . > > > >> > >> > > > > > > > > > > > > > > Using the NC_DISKLESS mode flag > > allows > > > >> for > > > >> > >> keeping > > > >> > >> > > > > > > > > > > > > > > the file in memory. The nc_open_mem > > > >> function > > > >> > >> allows > > > >> > >> > > > > for the > > > >> > >> > > > > > > > > > > > > > > read only access to treat a chunk of > > > >> memory > > > >> > >> as if > > > >> > >> > > it > > > >> > >> > > > > was > > > >> > >> > > > > > > > > > > > > > > a netcdf file. Unfortunately, for > > > >> netcdf-4, we > > > >> > >> > > > > ultimately > > > >> > >> > > > > > > > > > > > > > > depend on the HDF5 operation > > H5P_set_core > > > >> > >> > > > > > > > > > > > > > > (https://support.hdfgroup.org/ > > > >> > >> > > HDF5/doc/RM/RM_H5P.html# > > > >> > >> > > > > > > > > > > > > Property-SetFaplCore > > > >> > >> > > > > > > > > > > > > > > ). > > > >> > >> > > > > > > > > > > > > > > Upon close(), this can optionally > > store > > > >> the > > > >> > >> memory > > > >> > >> > > > > buffer in > > > >> > >> > > > > > > > a > > > >> > >> > > > > > > > > > > file. > > > >> > >> > > > > > > > > > > > > > > Unfortunately, it does not > > (currently) > > > >> > >> provide an > > > >> > >> > > > > option to > > > >> > >> > > > > > > > copy > > > >> > >> > > > > > > > > > > out > > > >> > >> > > > > > > > > > > > > the > > > >> > >> > > > > > > > > > > > > > > memory. > > > >> > >> > > > > > > > > > > > > > > The only solution I can see for now > > is to > > > >> > >> build an > > > >> > >> > > > > > > > alternative to > > > >> > >> > > > > > > > > > > the > > > >> > >> > > > > > > > > > > > > core > > > >> > >> > > > > > > > > > > > > > > driver > > > >> > >> > > > > > > > > > > > > > > that provides access (somehow) to the > > > >> memory . > > > >> > >> > > > > > > > > > > > > > > BTW, this is all going on at the > > netcdf-c > > > >> > >> library > > > >> > >> > > > > level. Our > > > >> > >> > > > > > > > pure > > > >> > >> > > > > > > > > > > jave > > > >> > >> > > > > > > > > > > > > HDF5 > > > >> > >> > > > > > > > > > > > > > > reader is read-only, hence cannot > > create > > > >> or > > > >> > >> modify > > > >> > >> > > > > files. > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > I have created an issue for this > > > >> > >> > > > > > > > > > > > > > > (https://github.com/Unidata/ne > > > >> > >> tcdf-c/issues/708) > > > >> > >> > > > > > > > > > > > > > > but it is not likely to get > > implemented > > > >> > >> anytime > > > >> > >> > > soon. > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > You will have to be content with > > writing > > > >> the > > > >> > >> > > contents > > > >> > >> > > > > to a > > > >> > >> > > > > > > > file. > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > I am using NetCDF Java library > > 4.6.10 . > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > My goal is to efficiently send > > > >> NetcdfFile > > > >> > >> > > > > > > > > > > > > > > > <https://www.unidata.ucar.edu/ > > > >> > >> > > > > software/thredds/current/ > > > >> > >> > > > > > > > > > > > > > > netcdf-java/javadoc/ucar/nc2/N > > > >> etcdfFile.html> > > > >> > >> > > > > > > > > > > > > > > > objects over the network using a > > > >> > >> communication > > > >> > >> > > > > library > > > >> > >> > > > > > > > (such as > > > >> > >> > > > > > > > > > > > > ZeroMQ) . > > > >> > >> > > > > > > > > > > > > > > > Because the NetcdfFile > > > >> > >> > > > > > > > > > > > > > > > <https://www.unidata.ucar.edu/ > > > >> > >> > > > > software/thredds/current/ > > > >> > >> > > > > > > > > > > > > > > netcdf-java/javadoc/ucar/nc2/N > > > >> etcdfFile.html> > > > >> > >> > > > > > > > > > > > > > > > class > > > >> > >> > > > > > > > > > > > > > > > does not implement Serializable, I > > > >> need to > > > >> > >> > > implement > > > >> > >> > > > > that > > > >> > >> > > > > > > > step > > > >> > >> > > > > > > > > > > > > myself. > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > Since NetCDF is a machine > > independent > > > >> data > > > >> > >> > > format, I > > > >> > >> > > > > would > > > >> > >> > > > > > > > like > > > >> > >> > > > > > > > > > > to > > > >> > >> > > > > > > > > > > > > access > > > >> > >> > > > > > > > > > > > > > > > the raw binary data. > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > This is trivial if the data kas > > been > > > >> > >> written to a > > > >> > >> > > > > file on > > > >> > >> > > > > > > > the > > > >> > >> > > > > > > > > > > disk. > > > >> > >> > > > > > > > > > > > > But, > > > >> > >> > > > > > > > > > > > > > > > what about in memory datasets ? If > > > >> > >> possible, I > > > >> > >> > > would > > > >> > >> > > > > like > > > >> > >> > > > > > > > to > > > >> > >> > > > > > > > > > > access > > > >> > >> > > > > > > > > > > > > to > > > >> > >> > > > > > > > > > > > > > > > binary data without writing it to > > > >> disk... > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > Can I access the buffer of an > > in-memory > > > >> > >> > > NetcdfFile > > > >> > >> > > > > > > > > > > > > > > > <https://www.unidata.ucar.edu/ > > > >> > >> > > > > software/thredds/current/ > > > >> > >> > > > > > > > > > > > > > > netcdf-java/javadoc/ucar/nc2/N > > > >> etcdfFile.html> > > > >> > >> > > > > > > > > > > > > > > > object > > > >> > >> > > > > > > > > > > > > > > > from the Java API ? Any pointers > > will > > > >> be > > > >> > >> > > appreciated. > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > Kind regards, > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > Michaël > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > =Dennis Heimbigner > > > >> > >> > > > > > > > > > > > > > > Unidata > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > Ticket Details > > > >> > >> > > > > > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > > > > > Ticket ID: MQO-415619 > > > >> > >> > > > > > > > > > > > > > > Department: Support netCDF Java > > > >> > >> > > > > > > > > > > > > > > Priority: Normal > > > >> > >> > > > > > > > > > > > > > > Status: Open > > > >> > >> > > > > > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > > > > > NOTE: All email exchanges with > > Unidata > > > >> User > > > >> > >> > > Support are > > > >> > >> > > > > > > > recorded > > > >> > >> > > > > > > > > > > in the > > > >> > >> > > > > > > > > > > > > > > Unidata inquiry tracking system and > > then > > > >> made > > > >> > >> > > publicly > > > >> > >> > > > > > > > available > > > >> > >> > > > > > > > > > > > > through > > > >> > >> > > > > > > > > > > > > > > the web. If you do not want to have > > your > > > >> > >> > > interactions > > > >> > >> > > > > made > > > >> > >> > > > > > > > > > > available > > > >> > >> > > > > > > > > > > > > in > > > >> > >> > > > > > > > > > > > > > > this way, you must let us know in > > each > > > >> email > > > >> > >> you > > > >> > >> > > send > > > >> > >> > > > > to us . > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > =Dennis Heimbigner > > > >> > >> > > > > > > > > > > > > Unidata > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > Ticket Details > > > >> > >> > > > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > > > Ticket ID: MQO-415619 > > > >> > >> > > > > > > > > > > > > Department: Support netCDF Java > > > >> > >> > > > > > > > > > > > > Priority: Normal > > > >> > >> > > > > > > > > > > > > Status: Open > > > >> > >> > > > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > > > NOTE: All email exchanges with Unidata > > User > > > >> > >> Support are > > > >> > >> > > > > recorded > > > >> > >> > > > > > > > in the > > > >> > >> > > > > > > > > > > > > Unidata inquiry tracking system and then > > made > > > >> > >> publicly > > > >> > >> > > > > available > > > >> > >> > > > > > > > > > > through > > > >> > >> > > > > > > > > > > > > the web. If you do not want to have your > > > >> > >> interactions > > > >> > >> > > made > > > >> > >> > > > > > > > available > > > >> > >> > > > > > > > > > > in > > > >> > >> > > > > > > > > > > > > this way, you must let us know in each > > email > > > >> you > > > >> > >> send > > > >> > >> > > to > > > >> > >> > > > > us. > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > =Dennis Heimbigner > > > >> > >> > > > > > > > > > > Unidata > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > Ticket Details > > > >> > >> > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > Ticket ID: MQO-415619 > > > >> > >> > > > > > > > > > > Department: Support netCDF Java > > > >> > >> > > > > > > > > > > Priority: Normal > > > >> > >> > > > > > > > > > > Status: Open > > > >> > >> > > > > > > > > > > =================== > > > >> > >> > > > > > > > > > > NOTE: All email exchanges with Unidata User > > > >> Support > > > >> > >> are > > > >> > >> > > > > recorded in > > > >> > >> > > > > > > > the > > > >> > >> > > > > > > > > > > Unidata inquiry tracking system and then made > > > >> publicly > > > >> > >> > > > > available > > > >> > >> > > > > > > > through > > > >> > >> > > > > > > > > > > the web. If you do not want to have your > > > >> > >> interactions made > > > >> > >> > > > > > > > available in > > > >> > >> > > > > > > > > > > this way, you must let us know in each email > > you > > > >> send > > > >> > >> to > > > >> > >> > > us. > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > =Dennis Heimbigner > > > >> > >> > > > > > > > > Unidata > > > >> > >> > > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > =Dennis Heimbigner > > > >> > >> > > > > > > > Unidata > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > Ticket Details > > > >> > >> > > > > > > > =================== > > > >> > >> > > > > > > > Ticket ID: MQO-415619 > > > >> > >> > > > > > > > Department: Support netCDF Java > > > >> > >> > > > > > > > Priority: Normal > > > >> > >> > > > > > > > Status: Open > > > >> > >> > > > > > > > =================== > > > >> > >> > > > > > > > NOTE: All email exchanges with Unidata User > > Support are > > > >> > >> recorded > > > >> > >> > > in > > > >> > >> > > > > the > > > >> > >> > > > > > > > Unidata inquiry tracking system and then made > > publicly > > > >> > >> available > > > >> > >> > > > > through > > > >> > >> > > > > > > > the web. If you do not want to have your > > interactions > > > >> made > > > >> > >> > > > > available in > > > >> > >> > > > > > > > this way, you must let us know in each email you > > send > > > >> to us. > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > > >> > >> > > > > > > > > >> > >> > > > > > =Dennis Heimbigner > > > >> > >> > > > > > Unidata > > > >> > >> > > > > > > > > >> > >> > > > > > > > >> > >> > > > > =Dennis Heimbigner > > > >> > >> > > > > Unidata > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > Ticket Details > > > >> > >> > > > > =================== > > > >> > >> > > > > Ticket ID: MQO-415619 > > > >> > >> > > > > Department: Support netCDF Java > > > >> > >> > > > > Priority: Normal > > > >> > >> > > > > Status: Open > > > >> > >> > > > > =================== > > > >> > >> > > > > NOTE: All email exchanges with Unidata User Support are > > > >> recorded > > > >> > >> in the > > > >> > >> > > > > Unidata inquiry tracking system and then made publicly > > > >> available > > > >> > >> > > through > > > >> > >> > > > > the web. If you do not want to have your interactions > > made > > > >> > >> available > > > >> > >> > > in > > > >> > >> > > > > this way, you must let us know in each email you send to > > us . > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > > >> > >> > > > > > > >> > >> > > > > > > >> > >> > > > > > >> > >> > > =Dennis Heimbigner > > > >> > >> > > Unidata > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > Ticket Details > > > >> > >> > > =================== > > > >> > >> > > Ticket ID: MQO-415619 > > > >> > >> > > Department: Support netCDF Java > > > >> > >> > > Priority: Normal > > > >> > >> > > Status: Closed > > > >> > >> > > =================== > > > >> > >> > > NOTE: All email exchanges with Unidata User Support are > > recorded > > > >> in > > > >> > >> the > > > >> > >> > > Unidata inquiry tracking system and then made publicly > > available > > > >> > >> through > > > >> > >> > > the web. If you do not want to have your interactions made > > > >> available > > > >> > >> in > > > >> > >> > > this way, you must let us know in each email you send to us . > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > > >> > >> > > > > >> > >> > > > > >> > >> > > > >> > >> =Dennis Heimbigner > > > >> > >> Unidata > > > >> > >> > > > >> > >> > > > >> > >> Ticket Details > > > >> > >> =================== > > > >> > >> Ticket ID: MQO-415619 > > > >> > >> Department: Support netCDF Java > > > >> > >> Priority: High > > > >> > >> Status: Open > > > >> > >> =================== > > > >> > >> NOTE: All email exchanges with Unidata User Support are recorded > > in > > > >> the > > > >> > >> Unidata inquiry tracking system and then made publicly available > > > >> through > > > >> > >> the web. If you do not want to have your interactions made > > > >> available in > > > >> > >> this way, you must let us know in each email you send to us. > > > >> > >> > > > >> > >> > > > >> > >> > > > >> > > > > > >> > > > > >> > > > > >> > > > >> =Dennis Heimbigner > > > >> Unidata > > > >> > > > >> > > > >> Ticket Details > > > >> =================== > > > >> Ticket ID: MQO-415619 > > > >> Department: Support netCDF Java > > > >> Priority: Critical > > > >> Status: Closed > > > >> =================== > > > >> NOTE: All email exchanges with Unidata User Support are recorded in > > the > > > >> Unidata inquiry tracking system and then made publicly available > > through > > > >> the web. If you do not want to have your interactions made available > > in > > > >> this way, you must let us know in each email you send to us. > > > >> > > > >> > > > >> > > > > > > > > > > > > > > =Dennis Heimbigner > > Unidata > > > > > > Ticket Details > > =================== > > Ticket ID: MQO-415619 > > Department: Support netCDF Java > > Priority: Critical > > Status: Open > > =================== > > NOTE: All email exchanges with Unidata User Support are recorded in the > > Unidata inquiry tracking system and then made publicly available through > > the web. If you do not want to have your interactions made available in > > this way, you must let us know in each email you send to us. > > > > > > > > =Dennis Heimbigner Unidata Ticket Details =================== Ticket ID: MQO-415619 Department: Support netCDF Java Priority: Critical Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.
Attachment:
inmemory.md
Description: Binary data