Torben,
PISM (www.pism-docs.org) uses NetCDF attributes to store the model
configuration[1]. This is not a great solution, but it served us well so
far. We store about 300 parameters, which, together with their metadata
(I had to come up with a homespun convention using attribute name
suffixes for this) gives around 1300 NetCDF variable attributes.
When using the NetCDF-3 file format you should pad the file header so
that all your attributes "fit" (see nc__enddef; note the double
underscore). In most cases this does not matter, but it could reduce the
time spent creating a NetCDF-3 file (especially if the file is large).
When using the HDF5-based NetCDF-4 file format the file header size is
not an issue, but *modifying* attributes is: see issue 350 on GitHub
[2]. It does not look like this issue will be fixed any time soon, so I
would advise against using attributes to store data that you may need to
update after the file is created.
As far as I know (hopefully NetCDF developers here will correct me if
I'm wrong) all file metadata is stored in RAM when a file is opened, so
accessing attributes is cheap *if* you keep the file open, but
increasing the number of attributes should increase the cost of
re-opening files. (This additional cost is not likely to be significant,
though.)
[1]: https://github.com/pism/pism/blob/master/src/pism_config.cdl
[2]: https://github.com/Unidata/netcdf-c/issues/350
On 11/30/17 7:06 AM, Helsloot, T. wrote:
Hello,
How bad of an idea is it to store data in attributes?
I want to store settings (metadata) containing 600+ key value pairs.
These settings may change in time, though infrequently and they don’t
need to be read often.
I could store them as attributes with a key and a 1D array of values,
which has the advantage that I have all the metadata within the .nc file.
In an extreme case the array will grow to a thousand entries.
Alternatively I would store the settings history in a separate text file.
Any thoughts on this are appreciated,
Regards,
Torben
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
--
Constantine
