NetCDF  4.7.0
inmemory.md
1 NetCDF In-Memory Support
2 ====================================
3 
4 <!-- double header is needed to workaround doxygen bug -->
5 
6 NetCDF In-Memory Support {#inmemory}
7 ====================================
8 
9 [TOC]
10 
11 Introduction {#inmemory_intro}
12 --------------
13 
14 It can be convenient to operate on a netcdf file whose
15 content is held in memory instead of in a disk file.
16 The netcdf API has been modified in a number of ways
17 to support this capability.
18 
19 Actually, three distinct but related capabilities are provided.
20 
21 1. DISKLESS -- Read a file into memory, operate on it, and optionally
22 write it back out to disk when nc_close() is called.
23 2. INMEMORY -- Tell the netcdf-c library to treat a provided block
24 of memory as if it were a netcdf file. At close, it is possible to ask
25 for the final contents of the memory chunk. Be warned that there is
26 some complexity to this as described below.
27 4. MMAP -- (deprecated) Tell the netcdf-c library to use the *mmap()* operating
28 system functionality to access a file.
29 
30 The first two capabilities are intertwined in the sense that the
31 *diskless* capability makes use internally of the *inmemory*
32 capability (for netcdf classic only). But, the *inmemory*
33 capability can be used independently of the *diskless*
34 capability.
35 
36 The *mmap()* capability provides a capability similar to *diskless* but
37 using special capabilities of the underlying operating system. It turns out
38 that the mmap capability has seen no significant use, so its use is deprecated
39 and will be removed at some point in the future.
40 
41 Note also that *diskless* and *inmemory* can be used for both
42 *netcdf-3* (classic) and *netcdf-4* (enhanced) data. The *mmap*
43 capability can only be used with *netcdf-3*.
44 
45 Enabling Diskless File Access {#Enable_Diskless}
46 --------------
47 The *diskless* capability can be used relatively transparently
48 using the *NC_DISKLESS* mode flag.
49 
50 Note that since the file is stored in memory, size limitations apply.
51 If you are on using a 32-bit pointer then the file size must be less than 2^32
52 bytes in length. On a 64-bit machine, the size must be less than 2^64 bytes.
53 
54 Also note that for a diskless file, there are two notions of
55 *write* with respect to the file. The first notion is that the
56 file is writeable through the netCDF API, but on disk, the file is
57 read-only. This means a call to, for example, _nc_def_dim()_ will succeed,
58 but no changes will be written to disk.
59 The second notion of *write* refers to the file on disk to which
60 the contents of memory might be persisted.
61 
62 WARNING: control of the two kinds of *write* has changed since
63 release 4.6.1.
64 
65 The mode flag NC_WRITE determines the first kind of *write*.
66 If set, then NC_WRITE means that the file can be modified through
67 the netCDF API, otherwise it is read-only. This is a change since
68 release 4.6.1.
69 
70 The new mode flag NC_PERSIST now determines the second kind of
71 *write*. If set, then NC_PERSIST means that the memory contents
72 will be persisted to disk, possibly overwriting the previous
73 file contents. Otherwise, the default is to throw away the
74 in-memory contents.
75 
76 ### Diskless File Open
77 Calling *nc_open()* using the mode flag *NC_DISKLESS* will cause
78 the file being opened to be read into memory. When calling *nc_close()*,
79 the file will optionally be re-written (aka "persisted") to disk. This
80 persist capability will be invoked if and only if *NC_PERSIST* is specified
81 in the mode flags at the call to *nc_open()*.
82 
83 ### Diskless File Create
84 Calling *nc_create()* using the mode flag *NC_DISKLESS* will cause
85 the file to initially be created and kept in memory.
86 When calling *nc_close()*, the file will be written
87 to disk if and only if *NC_PERSIST* is specified
88 in the mode flags at the call to *nc_create()*.
89 
90 Enabling Inmemory File Access {#Enable_Inmemory}
91 --------------
92 
93 The netcdf API has been extended to support the inmemory capability.
94 The relevant API is defined in the file `netcdf_mem.h`.
95 
96 The important data structure to use is `NC_memio`.
97 ````
98 typedef struct NC_memio {
99  size_t size;
100  void* memory;
101  int flags;
102 } NC_memio;
103 
104 ````
105 An instance of this data structure is used when providing or
106 retrieving a block of data. It specifies the memory and its size
107 and also some relevant flags that define how to manage the memory.
108 
109 Current only one flag is defined -- *NC_MEMIO_LOCKED*.
110 This tells the netcdf library that it should never try to
111 *realloc()* the memory nor to *free()* the memory. Note
112 that this does not mean that the memory cannot be modified, but
113 only that the modifications will be within the confines of the provided
114 memory. If doing such modifications is impossible without
115 reallocating the memory, then the modification will fail.
116 
117 ### In-Memory API
118 
119 The new API consists of the following functions.
120 ````
121 int nc_open_mem(const char* path, int mode, size_t size, void* memory, int* ncidp);
122 
123 int nc_create_mem(const char* path, int mode, size_t initialsize, int* ncidp);
124 
125 int nc_open_memio(const char* path, int mode, NC_memio* info, int* ncidp);
126 
127 int nc_close_memio(int ncid, NC_memio* info);
128 
129 ````
130 ### The **nc_open_mem** Function
131 
132 The *nc_open_mem()* function is actually a convenience
133 function that internally invokes *nc_open_memio()*.
134 It essentially provides simple read-only access to a chunk of memory
135 of some specified size.
136 
137 ### The **nc_open_memio** Function
138 
139 This function provides a more general read/write capability with respect
140 to a chunk of memory. It has a number of constraints and its
141 semantics are somewhat complex. This is primarily due to limitations
142 imposed by the underlying HDF5 library.
143 
144 The constraints are as follows.
145 
146 1. If the *NC_MEMIO_LOCKED* flag is set, then the netcdf library will
147 make no attempt to reallocate or free the provided memory.
148 If the caller invokes the *nc_close_memio()* function to retrieve the
149 final memory block, it should be the same
150 memory block as was provided when *nc_open_memio* was called.
151 Note that it is still possible to modify the in-memory file if the NC_WRITE
152 mode flag was set. However, failures can occur if an operation
153 cannot complete because the memory needs to be expanded.
154 2. If the *NC_MEMIO_LOCKED* flag is <b>not</b> set, then
155 the netcdf library will take control of the incoming memory.
156 This means that the user should not make any attempt to free
157 or even read the incoming memory block in this case.
158 The newcdf library is free to reallocate the incomming
159 memory block to obtain a larger block when an attempt to modify
160 the in-memory file requires more space. Note that implicit in this
161 is that the old block -- the one originally provided -- may be
162 free'd as a side effect of re-allocating the memory using the
163 *realloc()* function.
164 The caller may invoke the *nc_close_memio()* function to retrieve the
165 final memory block, which may not be the same as the originally block
166 provided by the caller. In any case, the returned block must always be freed
167 by the caller and the original block should not be freed.
168 
169 ### The **nc_create_mem** Function
170 
171 This function allows a user to create an in-memory file, write to it,
172 and then retrieve the final memory using *nc_close_memio()*.
173 The *initialsize* argument to *nc_create_mem()* tells the library
174 how much initial memory to allocate. Technically, this is advisory only
175 because it may be ignored by the underlying HDF5 library.
176 It is used, however, for netcdf-3 files.
177 
178 ### The **nc_close_memio** Function
179 
180 The ordinary *nc_close()* function can be called to close an in-memory file.
181 However, it is often desirable to obtain the final size and memory block
182 for the in-memory file when that file has been modified.
183 The *nc_close_memio()* function provides a means to do this.
184 Its second argument is a pointer to an *NC_memio* object
185 into which the final memory and size are stored. WARNING,
186 the returned memory is owned by the caller and so the caller
187 is responsible for calling *free()* on that returned memory.
188 
189 ### Support for Writing with *NC_MEMIO_LOCKED*
190 
191 When the NC_MEMIO_LOCKED flag is set in the *NC_memio* object
192 passed to *nc_open_memio()*, it is still possible to modify
193 the opened in-memory file (using the NC_WRITE mode flag).
194 
195 The big problem is that any changes must fit into the memory provided
196 by the caller via the *NC_memio* object. This problem can be
197 mitigated, however, by using the "trick" of overallocating
198 the caller supplied memory. That is, if the original file is, say, 300 bytes,
199 then it is possible to allocate, say, 65000 bytes and copy the original file
200 into the first 300 bytes of the larger memory block. This will allow
201 the netcdf-c library to add to the file up to that 65000 byte limit.
202 In this way, it is possible to avoid memory reallocation while still
203 allowing modifications to the file. You will still need to call
204 *nc_close_memio()* to obtain the size of the final, modified, file.
205 
206 Enabling MMAP File Access (Deprecated) {#Enable_MMAP}
207 --------------
208 
209 The MMAP functionality is deprecated.
210 
211 Some operating systems provide a capability called MMAP.
212 This allows disk files to automatically be mapped to chunks of memory.
213 It operates in a fashion somewhat similar to operating system virtual
214 memory, except with respect to a file.
215 
216 By setting mode flag NC_MMAP, it is possible to do the equivalent
217 of NC_DISKLESS but using the operating system's mmap capabilities.
218 
219 Currently, MMAP support is only available when using netcdf-3 or cdf5
220 files.
221 
222 Known Bugs {#Inmemory_Bugs}
223 --------------
224 
225 1. If you are modifying a locked memory chunk (using
226  NC_MEMIO_LOCKED) and are accessing it as a netcdf-4 file, and
227  you overrun the available space, then the HDF5 library will
228  fail with a segmentation fault.
229 
230 2. You will get an HDF5 error under the following conditions.
231 
232  1. You call nc_open on a file with the flags NC_DISKLESS|NC_WRITE
233  but without NC_PERSIST.
234  2. The file to be read is read-only (i.e. mode 0444).
235 
236  Note that this should be ok because the modifications to the file
237  are not intended to pushed back into the disk file. However, the
238  HDF5 core driver does not allow this.
239 
240 References {#Inmemory_References}
241 --------------
242 
243 1. https://support.hdfgroup.org/HDF5/doc1.8/Advanced/FileImageOperations/HDF5FileImageOperations.pdf
244 
245 Point of Contact
246 --------------
247 
248 __Author__: Dennis Heimbigner<br>
249 __Email__: dmh at ucar dot edu
250 __Initial Version__: 2/3/2018<br>
251 __Last Revised__: 2/5/2018
252 
253 

Return to the Main Unidata NetCDF page.
Generated on Mon Apr 29 2019 10:59:59 for NetCDF. NetCDF is a Unidata library.