There's been a proposal under CF for a similar construct, with lots of
discussion and objections, related to exactly the issue of the dataset
becoming useless if the associated file is missing.
http://cf-trac.llnl.gov/trac/ticket/145
The point is made there, that there are already plenty of "datasets"
that are in practice stored in multiple files, such as individual
timesteps, or different data fields that are stored separately. In those
cases, the files are valid files in and of themselves, but the model
run, or whatever it is, is contained in multiple files.
These kinds of proposals go further by storing dimension variables or
other information related to the grid in different files. Now if the
separately-stored information is missing or gets moved, interpreting the
file contents becomes impossible.
-Ansley
On 7/13/2016 2:40 PM, Roy Mendelssohn - NOAA Federal wrote:
that if one of those underlying files is missing,
then I assume the whole dataset is unreadable.
I have been wondering that myself. Is there anyone from HDF that can answer
that?
-Roy
On Jul 13, 2016, at 2:33 PM, dmh@xxxxxxxx wrote:
If I understand it correctly, you are correct: from the API
point of view, it is invisible. But that is not the issue for me.
The issue is that now one needs multiple files in order to acccess
data rather than having everything kept in a single file. That means,
for example, that if one of those underlying files is missing,
then I assume the whole dataset is unreadable.
=Dennis Heimbigner
Unidata
On 7/13/2016 3:25 PM, Roy Mendelssohn - NOAA Federal wrote:
Hi Dennis:
I asked about this also awhile back. The key question to my mind is how the
virtual dataset works in HDF5. Perhaps the interface is such is that it is
transparent to higher level calls, much like when you create virtual datasets
in TDS. In that case, the calls would be to as if there was one file.
-Roy
On Jul 13, 2016, at 2:09 PM, dmh@xxxxxxxx wrote:
To this point in time, it has been an implicit assumption
that a netcdf dataset encompasses a single file. This functionality
effectively challenges that assumption. So, I would solicit
community input on this. What problems are created if a netcdf
dataset now requires multiple files?
=Dennis Heimmbigner
Unidata
On 7/13/2016 5:09 AM, Eugen Betke wrote:
Dear NetCDF-Group,
we have been working on NetCDF external link functionality. This allows
NetCDF applications to create dimension variables which values are
stored in an external file. Therefore, it uses the HDF5 virtual dataset
(VDS) functionality. This is useful for, e.g., climate applications that
rely on a variable per file and timestep configuration. The idea is to
store the grid in a separate file and link our data to this grid. We
already have our first working version. You find the patch and the
examples on our page:
http://wr.informatik.uni-hamburg.de/research/projects/bullio/netcdf_external_links/start
Under the hood it uses HDF5 virtual datasets. VDS has the advantage of
being compatible to the functions that are supported by oridinary
datasets. Therefore, files containing VDS should be supported by the
most software.
There is a minor issue related to HDF5, the call H5F_try_close function
fails, when ncdump trys to read data from an external dimension. So far
we found a workaround, but we will fix this issue.
It would be great if external link functionality could be supported by
netCDF at some timepoint. We would like to improve our patch and for
that reason we need your feedback. If you have some idea to the issue
above, we would be grateful for each hint.
Regards,
Eugen
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
**********************
"The contents of this message do not reflect any position of the U.S. Government or
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
**********************
"The contents of this message do not reflect any position of the U.S. Government or
NOAA."
**********************
Roy Mendelssohn
Supervisory Operations Research Analyst
NOAA/NMFS
Environmental Research Division
Southwest Fisheries Science Center
***Note new address and phone***
110 Shaffer Road
Santa Cruz, CA 95060
Phone: (831)-420-3666
Fax: (831) 420-3980
e-mail: Roy.Mendelssohn@xxxxxxxx www: http://www.pfeg.noaa.gov/
"Old age and treachery will overcome youth and skill."
"From those who have been given much, much will be expected"
"the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
