[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #RTA-885842]: Writing several netcdf files simultaneously by multiple threads

Hi Christian,

Ah, I see, I’m sorry; I misunderstood when answering your original
question, my apologies!

Unfortunately, this isn’t possible; there is no way to guarantee that
netCDF data l is stored in contiguous memory; this is (as I understand it)
why the netCDF API uses an ncid identifier instead of a pointer to a file
handle. What you’re doing is probably your best bet using current netCDF,
as it lets you read the file in as a byte stream into a contiguous block of
memory.

This conversation does make me wonder if there’s a need for something along
the lines of nc_serialize_to_mem(); instead of writing to disk (which is
still not guaranteed contiguous), write into a contiguous block of memory.
I’ll need to think about this some but it seems like something that might
be useful.

-Ward

address@hidden> wrote:

New Client Reply: Writing several netcdf files simultaneously by multiple
> threads
>
> Hi Ward and Dennis
>
> @Ward:
> I’m right now looking into nc_open_mem(), but I’m not sure whether I do
> not understand correctly, or whether we misunderstood each other.
> As I understand it nc_open_mem() can be used to open a netcdf which is
> already in a block of memory. To do whatever you would do with it, and then
> e.g. write it to a file again.
> My case however is a bit different: I want to open a block of memory (whom
> I don’t know the size of yet), and want to use the nc_put…() methods to
> write into that memory. Then instead of writing the netcdf to a file, I
> want to transfer the block of memory, containing a full netcdf now, via
> sockets.
> So the question is actually not how to open a block of memory containing a
> netcdf, but how to write a netcdf into a block of memory.
>
> What’ I’m doing now is this:
>
> nc_create(unique_filename,…);
> nc_put_att_text(…)
> …
> nc_enddef(…)
> …
> nc_put_var_double(…)
> …
> nc_close(…)
>
> // so now I have a netcdf file with my content
> // then I read the file again and write it into a buffer:
> ifstream Rein(unique_filename.c_str());
> const int bytesToLoad=get_file_size(unique_filename);
> if (length(buffer) != bytesToLoad)
> {
>      buffer.Alloc(bytesToLoad);
> }
> Rein.read(&buffer[0], bytesToLoad);
> Rein.close();
> std::remove(unique_filename.c_str());
>
> // then I can send the buffer
>
>
> Obviously, the disk-IO involved in that is completely useless. Therefore I
> want to get rid of the file. Can nc_open_mem indeed be used for this?
>
>
> @Dennis:
> Thanks for your information. I’m definitely looking forward to
> multithreading support. I’m currently locking the accesses to netcdf, but
> that will eventually lead to performance problems.
>
> Best regards,
> Christian
>
>
>
>
> address@hidden> wrote:
> >
> > Hello Christian,
> >
> > It is possible to do what you're describing; you will need to pass a
> non-null 'path' to specify the dataset filename, but other than that you
> will not need to worry about any disk I/O.
> >
> > The function you are interested in is `nc_open_mem()`, and the
> documentation can be found here:
> >
> > * http://www.unidata.ucar.edu/software/netcdf/docs/group__datasets.html#
> gac12fdf7579a2619b2aeb238cea2e7377
> >
> > Regarding thread safety; netCDF is not thread safe, as you've
> discovered; this is on our list of upcoming features, but it is not
> available in the current release.
> >
> > Sorry I can't provide a more immediately useful solution for the second
> question, but hopefully you will be able to use nc_open_mem() for your
> purposes.
> >
> > Thanks!
> >
> > -Ward
> >> Hi there
> >>
> >> I have two questions regarding the netcdf c library and would highly
> >> appreciate a quick answer.
> >>
> >> We want to generate netcdf files upon user requests and deliver the file
> >> via a Rest-API. At the moment we do that by writing a file to disk with
> >> the netcdf C library and then read the file again to send it to the
> >> user. We would prefer directly generating the netcdf into a memory
> >> location or a stream, so that we do not need to do the detour via
> >> writing an actual file to disk. Is there a way to do that?
> >> Unfortunately, I couldn’t find any netcdf-write-methods that do
> >> not work with filenames but with streams, pointers, o.s. instead.
> >>
> >> The second question is regarding thread safety. I read all the
> >> documentation about it I could find, but couldn’t find a
> >> solution to this specific case: We get segmentation faults when multiple
> >> independent files (different data, different filenames) are written by
> >> multiple threads at the same time (one file is written by each thread) .
> >> I did build  HDF5 with the enable-threadsafe option and used the
> >> manually built HDF5 when compiling netcdf from source.
> >> I did not however enable parallel-IO, but if I understand it right, this
> >> is not necessary (or even worse) in our use case, as we do not want to
> >> write one single file in a multi-threaded way, but several independent
> >> files.
> >>
> >> Thank you very much in advance and best regards,
> >> Christian Schluchter
> >>
> >>
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: RTA-885842
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the
> Unidata inquiry tracking system and then made publicly available through
> the web.  If you do not want to have your interactions made available in
> this way, you must let us know in each email you send to us.
> >
> >
>
>
>
>
> Ticket Details
> ===================
> Ticket ID: RTA-885842
> Department: Support netCDF
> Priority: Normal
> Status: Open
> Link:  https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=
> tickets&_a=viewticket&ticketid=27561
>
> ​



Ticket Details
===================
Ticket ID: RTA-885842
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.