[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20020605: 2GB file size limit.

>To: address@hidden
>From: Jacques Middlecoff <address@hidden>
>Subject: 2GB file size limit.
>Organization: UCAR/Unidata
>Keywords: 200206052234.g55MYLJ18609

Hi Jacques,

> I'm using netCDF 3.4 and have run into the 2GB file size limit.  In my
> manual, version 3 June 1997, it just says that 2GB is the limit, but on
> your website it says that if the platform has "Large File Support" (LFS)
> then it's possible to write files larger than 2GB by following certain
> procedures.  I tried it and I couldn't get it to write files larger than
> 2GB.  Our platform, jet an alpha cluster, supports files larger than 2GB.
> What is LFS?  Is it related to the machine or netCDF or what?  Do I need a
> later version of netCDF?

"Large File Support" (LFS) is a standard for Unix systems that permits
accessing files that are larger than 2 GBytes:

  http://ftp.sas.com/standards/large.file/

Most Unix platforms now provide LFS, even on 32-bit platforms, such as
Linux on Intel, though for these you have to compile with a special
macro defined to get LFS support.   But on an alpha or any other
64-bit platform, you shouldn't need to do anything special to get LFS.

However, there are some constraints on the structure of large netCDF
files that result from the 32-bit relative offsets that are part of
the netCDF file format.  In order to get netCDF files larger than 2
GBytes, you have to follow these constraints:

  
http://www.unidata.ucar.edu/packages/netcdf/f90/Documentation/f90-html-docs/guide9.html#2236524

I've appended an example C program that should write a 3Gbyte netCDF
file.  You can adjust the size by changing the definition of the NRECS
macro.  This should work for netCDF 3.4 as well as netCDF 3.5, but not
for earlier versions.  I don't have access to an Alpha to test this on
currently, so I'd like to know if you compile and run this and it
doesn't work.

Just put the program in a file named "big.c" and compiled and run it like
this:

 $ c89 -o big -I/home/russ/netcdf/include big.c -L/home/russ/netcdf/lib -lnetcdf
 $ ./big

--Russ

#include <stdio.h>
#include <stdlib.h>
#include <netcdf.h>

void
check_err(const int stat, const int line, const char *file) {
    if (stat != NC_NOERR) {
           (void) fprintf(stderr, "line %d of %s: %s\n", line, file, 
nc_strerror(stat));
        exit(1);
    }
}

#define NFIXED 1000000
#define NRECS  750 /* 100 -> 400 MB, 500 -> 2GB, 750 -> 3GB */

int
main() {                        /* create big.nc */

   int  ncid;                   /* netCDF id */

   /* dimension ids */
   int n_dim;
   int r_dim;

   /* dimension lengths */
   size_t n_len = NFIXED;
   size_t r_len = NC_UNLIMITED;

   /* variable ids */
   int x_id;

   /* rank (number of dimensions) for each variable */
#  define RANK_x 2

   /* variable shapes */
   int x_dims[RANK_x];

   /* enter define mode */
   int stat = nc_create("big.nc", NC_CLOBBER, &ncid);
   check_err(stat,__LINE__,__FILE__);

   /* define dimensions */
   stat = nc_def_dim(ncid, "n", n_len, &n_dim);
   check_err(stat,__LINE__,__FILE__);
   stat = nc_def_dim(ncid, "r", r_len, &r_dim);
   check_err(stat,__LINE__,__FILE__);

   /* define variables */

   x_dims[0] = r_dim;
   x_dims[1] = n_dim;
   stat = nc_def_var(ncid, "x", NC_FLOAT, RANK_x, x_dims, &x_id);
   check_err(stat,__LINE__,__FILE__);

   /* leave define mode */
   stat = nc_enddef (ncid);
   check_err(stat,__LINE__,__FILE__);

   {                    /* store x */
    static size_t x_start[RANK_x];
    static size_t x_count[RANK_x];
    float x[NFIXED];
    int i;
    int r;

    r_len = NRECS;                      /* number of records of x data */
    
    for (i=0; i<NFIXED; i++) {
      x[i] = i;
    }    
    
    for (r = 0; r < NRECS; r++) {
      x_start[0] = r;
      x_start[1] = 0;
      x_count[0] = 1;
      x_count[1] = n_len;
      
      stat = nc_put_vara_float(ncid, x_id, x_start, x_count, x);
      check_err(stat,__LINE__,__FILE__);
    }
   }
   stat = nc_close(ncid);
   check_err(stat,__LINE__,__FILE__);
   return 0;
}