[netcdfgroup] Possible bug when reading HDF5 files with "fixed length String" data type

• To: "netcdfgroup@xxxxxxxxxxxxxxxx List" <netcdfgroup@xxxxxxxxxxxxxxxx>
• Subject: [netcdfgroup] Possible bug when reading HDF5 files with "fixed length String" data type
• From: John Caron <jcaron1129@xxxxxxxxx>
• Date: Mon, 6 Mar 2023 19:41:55 -0700

Hello friends:

The attached code shows a possible bug in the netcdf C library when reading
"fixed length String" data type in HDF5 files, by not zero terminating
them. The output of that program is:

./testClib
*** nc_inq_grp_ncid =65537
*** nc_inq_varid =0
*** nc_inq_att type=12 attlen = 1
   CAPE:units[0] = 'J/kg�U'

The correct answer should be

   CAPE:units[0] = 'J/kg'

I think the problem might be in hdf5open.c, line 1904:

/* Copy strings, one at a time, into their new home. Alloc
  space for each string. The user will later free this
   space with nc_free_string. */

for (i = 0; i < att->len; i++)
{
    if (!(att->stdata[i] = malloc(fixed_size))) {
        free(contig_buf);
        BAIL(NC_ENOMEM);
    }
    strncpy(att->stdata[i], cur, fixed_size);
    cur += fixed_size;
}

This probably should be (note mallocing an extra char to be sure the string
is zero terminated):

for (i = 0; i < att->len; i++)
{
    if (!(att->stdata[i] = malloc(fixed_size+1))) {
        free(contig_buf);
        BAIL(NC_ENOMEM);
    }
    strncpy(att->stdata[i], cur, fixed_size);
    cur += fixed_size;
}

IIUC, these strings are returned as char *, and theres no way to find out
their length from the netcdf API. So they have to be zero terminated.
Otherwise the reading code keeps reading into unallocated memory, and only
stops reading when a zero byte happens to be found.

>From comments in the code, it would appear that this type is not used in
files written with the  netcdf library. If so, it only affects HDF5 files
not written with netcdf library, that use "fixed length String" data types.

Regards,
John

/*
export LD_LIBRARY_PATH="/usr/lib/x86_64-linux-gnu"
gcc -o testClib testClib.c -lnetcdf
*/

#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <netcdf.h>

/* This is the name of the data file we will read. */
#define FILE_NAME 
"/media/twobee/netch/joleenf/IASI_20120229022657Z.atm_prof_rtv.h5"

/* We are reading 2D data, a 6 x 12 grid. */
#define NX 6
#define NY 12

/* Handle errors by printing an error message and exiting with a
 * non-zero status. */
#define ERRCODE 2
#define ERR(e) {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

int
main()
{
   /* This will be the netCDF ID for the file and data variable. */
   int ncid, grpid, varid;

   /* Loop indexes, and error handling. */
   int retval;

   /* Open the file. NC_NOWRITE tells netCDF we want read-only access
    * to the file.*/
   if ((retval = nc_open(FILE_NAME, NC_NOWRITE, &ncid)))
      ERR(retval);

   /* Get the grpid, based on its name. */
   if ((retval = nc_inq_grp_ncid(ncid, "All_Data", &grpid)))
      ERR(retval);
    printf("*** nc_inq_grp_ncid =%d\n", grpid);

   /* Get the varid of the data variable, based on its name. */
   if ((retval = nc_inq_varid(grpid, "CAPE", &varid)))
      ERR(retval);
    printf("*** nc_inq_varid =%d\n", varid);

   /* get the attribute unit. */
   int type;
   long attlen;
   if ((retval = nc_inq_att(grpid, varid, "units", &type, &attlen)))
      ERR(retval);
   printf("*** nc_inq_att type=%d attlen = %ld\n", type, attlen);
   /* expect type = 12, size = 1 */

   char **string_attr = (char**)malloc(attlen * sizeof(char*));
   memset(string_attr, 0, attlen * sizeof(char*));

   if ((retval = nc_get_att_string(grpid, varid, "units", string_attr)))
      ERR(retval);
   for (size_t k = 0; k < attlen; ++k) {
     printf("   CAPE:units[%ld] = '%s'\n", k, string_attr[k]);
   }

   if ((retval = nc_free_string(attlen, string_attr)))
        ERR(retval);
   free(string_attr);

   /* Close the file, freeing all resources. */
   if ((retval = nc_close(ncid)))
      ERR(retval);

   printf("*** SUCCESS reading example file %s!\n", FILE_NAME);
   return 0;
}

Attachment: IASI_20120229022657Z.atm_prof_rtv.h5.7z
Description: application/7z-compressed

• Follow-Ups:
  • Re: [netcdfgroup] Possible bug when reading HDF5 files with "fixed length String" data type
    • From: Dennis Heimbigner