[netcdfgroup] Announcement: procedural generation of datasets with Lua

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] Announcement: procedural generation of datasets with Lua
• From: "Lucas Villa Real" <lucasvr@xxxxxxxxxx>
• Date: Sat, 20 Jun 2020 03:52:20 +0000
• Sensitivity:

<div class="socmaildefaultfont" dir="ltr" style="font-family:Arial, Helvetica, 
sans-serif;font-size:10pt" ><div dir="ltr" ><div><p>On behalf of my team, I’m 
happy to announce the first release of HDF5-UDF: user-defined-functions for 
HDF5 (and NetCDF4, by extension). The project enables the embedding of Lua 
scripts in HDF5 so that users can programmatically define a dataset whose data 
is generated on-the-fly each time that dataset is read.</p>
<p>The primary motivation for this project is to dramatically reduce the disk 
space used by datasets that are a variation of existing data. We have 
successfully used HDF-UDF to virtually eliminate the impact of derived data in 
a number of use cases; grids that used to take a few gigabytes on disk, 
uncompressed, now require just a couple of kilobytes.</p>
<p>Underneath, the source code is converted to a bytecode representation that 
LuaJIT executes when the dataset is read by the application. Through 
Just-In-Time compilation the overhead of virtualization is barely noticed: 
outputting grids that have no dependency on existing datasets can be an order 
of magnitude faster than reaching out to disk for I/O.</p>
<div>HDF5-UDF is shipped as a filter, which means that no modifications are 
needed to applications that read from NetCDF4 files.</div>
<div>&nbsp;</div>
<div>The project is hosted at <a href="https://github.com/lucasvr/hdf5-udf"; 
>https://github.com/lucasvr/hdf5-udf</a></div>
<div>&nbsp;</div>
<div>We invite everyone to try it out and to open pull requests. We hope you 
find it as useful as we do.</div>
<p>Thanks,<br>Lucas</p></div></div></div><BR>