Re: no chunking or deflate for scalar values in HDF5?

Hi Ed,

On Jul 12, 2007, at 6:42 AM, Ed Hartnett wrote:

Howdy all HDF5ers!

I have been asked by a user whether the zlib deflate can be used on
scalar datasets.

I believe from my experiments in HDF5 that it cannot, because there
is no way to set the chunking for a scalar. Is this correct?

I don't believe this is discussed in the reference manual - am I
missing it?

The code below, for example, fails when I try to set the chunking.

Is this the correct conclusion, or am I missing something?

Interesting - not something I thought about trying, since it's not very useful in general. I suppose it could be beneficial if the datatype for the dataset was large enough that compression saved some space (although those gains would be offset by the B-tree overhead for indexing the chunks).

It is not currently supported, since a scalar dataspace corresponds to a 0-dimensional "point", and chunks must have dimensions >=1. If this is critically important, I can work on the modifications to the code to allow this to work, but it would require changes in a number of places and I'd rather not go that direction right now... :-)

        Quincey

Thanks!

Ed

   printf("*** Checking HDF5 scalar variable compession...");

#define MAX_NAME_LEN 50
#define DEFLATE_LEVEL 3
#define SIMPLE_VAR_NAME1 "punches"
   {
      hid_t fapl_id, fcpl_id;
      hid_t datasetid;
      hid_t fileid, grpid, spaceid, plistid;
      int data_in, data_out = 42;
      hsize_t chunksize = 1;

      /* Create file, setting latest_format in access propertly list
       * and H5P_CRT_ORDER_TRACKED in the creation property list. */
      if ((fapl_id = H5Pcreate(H5P_FILE_ACCESS)) < 0) ERR;
      if (H5Pset_latest_format(fapl_id, 1) < 0) ERR;
      if ((fcpl_id = H5Pcreate(H5P_FILE_CREATE)) < 0) ERR;
if (H5Pset_link_creation_order(fcpl_id, H5P_CRT_ORDER_TRACKED| H5P_CRT_ORDER_INDEXED) < 0) ERR; if ((fileid = H5Fcreate(FILE_NAME, H5F_ACC_TRUNC, fcpl_id, fapl_id)) < 0) ERR;

      if ((grpid = H5Gopen(fileid, "/")) < 0) ERR;

      if ((spaceid = H5Screate(H5S_SCALAR)) < 0) ERR;

      /* Create property lust. */
      if ((plistid = H5Pcreate(H5P_DATASET_CREATE)) < 0) ERR;

      if (H5Pset_chunk(plistid, 1, &chunksize) < 0)ERR;

      /* Set up compression. */
      if (H5Pset_deflate(plistid, DEFLATE_LEVEL) < 0) ERR;

      /* Create the variable. */
if ((datasetid = H5Dcreate(grpid, SIMPLE_VAR_NAME1, H5T_NATIVE_INT,
                                 spaceid, plistid)) < 0) ERR;

      /* Write the data. */
      if (H5Dwrite(datasetid, H5T_NATIVE_INT, H5S_ALL, H5S_ALL,
                   H5P_DEFAULT, &data_out) < 0) ERR;

      if (H5Dclose(datasetid) < 0) ERR;
      if (H5Pclose(fapl_id) < 0 ||
          H5Sclose(spaceid) < 0 ||
          H5Gclose(grpid) < 0 ||
          H5Fclose(fileid) < 0)
         ERR;

      /* Now reopen the file and check. */
      if ((fapl_id = H5Pcreate(H5P_FILE_ACCESS)) < 0) ERR;
      if (H5Pset_latest_format(fapl_id, 1) < 0) ERR;
if ((fileid = H5Fopen(FILE_NAME, H5F_ACC_RDONLY, fapl_id)) < 0) ERR;
      if ((grpid = H5Gopen(fileid, "/")) < 0) ERR;

      if ((datasetid = H5Dopen(grpid, SIMPLE_VAR_NAME1)) < 0) ERR;
      if ((spaceid = H5Dget_space(datasetid)) < 0)
      if (H5Dread(datasetid, H5T_NATIVE_INT, H5S_ALL,
                  spaceid, H5P_DEFAULT, &data_in) < 0) ERR;

      /* Check the data. */
      if (data_in != data_out) ERR;

      if (H5Pclose(fapl_id) < 0 ||
          H5Dclose(datasetid) < 0 ||
          H5Gclose(grpid) < 0 ||
          H5Fclose(fileid) < 0)
         ERR;
   }
   SUMMARIZE_ERR;


--
Ed Hartnett  -- ed@xxxxxxxxxxxxxxxx

====================================================================== ========
To unsubscribe netcdf-hdf, visit:
http://www.unidata.ucar.edu/mailing-list-delete-form.html
====================================================================== ========



Attachment: smime.p7s
Description: S/MIME cryptographic signature