[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020605: 2GB file size limit.

>To: address@hidden
>From: Jacques Middlecoff <address@hidden>
>Subject: 2GB file size limit.
>Organization: UCAR/Unidata
>Keywords: 200206052234.g55MYLJ18609

Jacques,

> I ran big.c on jet and got the same error as I did with my Fortran
> test case when it ran into the 2BG limit:
> 
>  204>  ls -lrt | tail
> -rw-------   1 jacquesm ad             76 Jun  5 15:04 qsub1.o252745
> -rw-------   1 jacquesm ad              0 Jun  5 15:04 qsub1.e252745
> -rw-r--r--   1 jacquesm ad            993 Jun  5 15:31 testnetCDFw4.f
> -rw-r--r--   1 jacquesm ad           1988 Jun  6 11:31 big.c
> -rwxr-xr-x   1 jacquesm ad         466770 Jun  6 11:32 big*
> -rw-r--r--   1 jacquesm ad       2147475456 Jun  6 11:47 big.nc
> -rw-------   1 jacquesm ad        4210688 Jun  6 11:47 core
> -rw-------   1 jacquesm ad            255 Jun  6 11:47 qsub1.o273069
> -rw-------   1 jacquesm ad            128 Jun  6 11:47 qsub1.e273069
>  205>  cat qsub1.o273069
> Here is the prologue
> prologue: nodefile is /usr/local/pbs/aux/273069.aileron
> Warning: no access to tty (Bad file descriptor).
> Thus no job control in this shell.
> In interactive mode.
> xterm
> Abort (core dumped)
>  206>  cat qsub1.e273069
> big: posixio.c:273: px_get: Assertion `offset < 2147483647' failed.

Oops, sorry, I should have remembered this problem, because I just
recently saw it for another user.  It turns out that the C library
must be compiled with the -DNDEBUG flag in order to permit writing
large files, because of an incorrect assert() statement left in for
debugging purposes around line 277 in netCDF 3.5.0 (line 273 in netCDF
3.4) of libsrc/posixio.c:

        assert(offset < X_INT_MAX); /* sanity check */

Compiling with -DNDEBUG turns off assertion checking and allows large
files to be written.  Alternatively you can just delete the above
statement and recompile the library.

We have also fixed occurrences of these too-strict assertions in the
beta version netcdf-3.5.1-beta3.tar.Z available from

  ftp://ftp.unidata.ucar.edu/pub/netcdf/netcdf-beta.tar.Z

--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu