[netcdfgroup] Odd failure reading compressed 32-bit floats with nc_get_var_double()?

• To: netcdfgroup@xxxxxxxxxxxxxxxx
• Subject: [netcdfgroup] Odd failure reading compressed 32-bit floats with nc_get_var_double()?
• From: David Pierce <dpierce@xxxxxxxx>
• Date: Tue, 8 Sep 2020 15:30:52 -0700
• Pp_allow_relay: proofpoint_allowed

Hello netcdf-ers,

I had a user of the R ncdf4 package alert me to an odd and perplexing
apparent bug in the netcdf library. It is triggered by the netcdf file you
can download here:

http://cirrus.ucsd.edu/~pierce/tmp/mendota_buoy.2018-11-08.nc

This file has the following variable (among others):

float phycocyanin(time) ;
            phycocyanin:_FillValue = -9999.f ;
            phycocyanin:units = "RFU" ;
            phycocyanin:long_name = "Phycocyanin" ;
            phycocyanin:_Storage = "chunked" ;
            phycocyanin:_ChunkSizes = 1440 ;
            phycocyanin:_DeflateLevel = 4 ;
            phycocyanin:_Shuffle = "true" ;
            phycocyanin:_Endianness = "little" ;

'ncdump' reports all the data as missing:

phycocyanin = _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
_, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _, _,
(etc.)

When I read the data into a floating point array using the C library using
nc_get_var_float() then I get the expected values of -9999.0. However when
I read it into a double precision array with nc_get_var_double() then I get
strange values. Here is the output of the little test program that I've
appended below:

Sizeof float: 4 double: 8
======== DOUBLE ========
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
-559239646634519513659653226496.000000
======== FLOAT ========
-9999.000000 -9999.000000 -9999.000000 -9999.000000 -9999.000000
-9999.000000 -9999.000000 -9999.000000 -9999.000000 -9999.000000

This seems like a bug, unless I'm overlooking something obvious in the test
code (always possible). The nc_get_var_double() call is supposed to convert
the netcdf file's values into double precision and store them in the
provided double precision array, right? That's my understanding anyway. I
can't see why they are not -9999.00's in the resultant array.

Any thoughts are appreciated,

--Dave

Test program:

#include <stdio.h>
#include <stdlib.h>
#include "netcdf.h"

void main( int argc, char *argv[] )
{
        int     err, ncid, varid;
        size_t  nt;
        double  *ddat;
        float   *fdat;

        nt = 1440;      /* just hardcode for test */

        printf( "Sizeof float: %ld  double: %ld\n", sizeof(float),
sizeof(double) );

        err = nc_open("mendota_buoy.2018-11-08.nc", 0, &ncid );
        if( err != 0 ) {
                printf( "err open = %d\n", err );
                exit(-1);
                }

        err = nc_inq_varid( ncid, "phycocyanin", &varid );
        if( err != 0 ) {
                printf( "err inq_varid = %d\n", err );
                exit(-1);
                }

        /* Make room for both double and floating dat */
        ddat = (double *)malloc( sizeof(double) * nt );
        fdat = (float  *)malloc( sizeof(float ) * nt );

        /* Read into double array. Supposed to convert to double...? */
        err = nc_get_var_double( ncid, varid, ddat );
        if( err != 0 ) {
                printf( "err get_var_double = %d\n", err );
                exit(-1);
                }

        printf( "======== DOUBLE ========\n" );
        for( int ii=0; ii<10; ii++ )
                printf( "%lf  ", ddat[ii] );

        printf( "\n" );

        /* === Do same thing, but for float === */

        err = nc_get_var_float( ncid, varid, fdat );
        if( err != 0 ) {
                printf( "err get_var_float = %d\n", err );
                exit(-1);
                }

        printf( "======== FLOAT ========\n" );
        for( int ii=0; ii<10; ii++ )
                printf( "%f  ", fdat[ii] );

        printf( "\n" );

}


-------------------------------------------------------------------
David W. Pierce
Division of Climate, Atmospheric Science, and Physical Oceanography
Scripps Institution of Oceanography
(858) 534-8276 (voice)  /  (858) 534-8561 (fax)    dpierce@xxxxxxxx
-------------------------------------------------------------------

• Follow-Ups:
  • Re: [netcdfgroup] Odd failure reading compressed 32-bit floats with nc_get_var_double()?
    • From: Sylvain Herlédan