[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ZCE-849683]: clarification on large file support

> I did as you suggested me.
> 
> But I get the following outcome:
> 
> Size of Size_t = 8 ok
> Size of Off_t = 4  !unbelievable
> 
> How does it come?
> We expect both of quantities to be equal to 8 bytes on a 64 bit platform,
> don't they?

I'm surprised that the default size of off_t is only 4, but maybe
you need to set a C macro to get an 8-byte off_t.  Try compiling 
and running this C program:

#define _FILE_OFFSET_BITS 64
#include <sys/types.h>
#include <stdio.h>
int main(void) {
printf("Size of off_t is %d bytes.\n", sizeof(off_t));
printf("Size of size_t is %d bytes.\n", sizeof(size_t));
return 0;
}

If that shows sizeof(off_t) is now 8 bytes, then you can rebuild
netCDF setting the CFLAGS environment variable, for example

  CFLAGS='-D_FILE_OFFSET_BITS=64'

> Maybe some particular preprocessor have to be defined before compiling the
> netcdf lib?
> I can find the following preprocessors in my solution:
> _WINDOWS;_USRDLL;NETCDF_EXPORTS;VERSION=3.6.1-beta1;DLL_EXPORT;VISUAL_CPLUSP
> LUS;_FILE_OFFSET_BITS=64
> 
> I removed the NETCDF_DLL preprocessor since I need to create a static
> library rather then a dynamic one.
> 
> Can you give me some more suggestions to get the library work properly with
> large file support?

I wish I knew more about the Windows compilation environment for
64-bit support.  We may get these problems solved in the next six
months with our planned port to Windows that was recently announced
on the netcdfgroup mailing list:

  
http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00257.html

--Russ

> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: Monday, July 19, 2010 6:15 PM
> To: address@hidden
> Cc: address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: [netCDF #ZCE-849683]: clarification on large file support
> 
> Fabio,
> 
> > Please, I need just one more clarification.
> > I would like to use the netcdf library on windows 64 bit platform.
> > In order to activate very large file support, is it sufficient to
> > compile the library specifying a 64bit environment and activate the
> > 64bit file offset. Doing so, does the size_t type automatically switch
> > to a 64 bit unsigned integer variable in such a way not to limit
> > memory and thus data set allocation. I am using your code and Visual
> > studio 2008 and I'm specifing a 64 bit target environment.
> 
> With a 64-bit development environment, there is no need to "activate the
> 64bit file offset", and this depends on the off_t type, not size_t.
> 
> All versions of netCDF since 3.6.0 include support for reading and writing
> 64-bit offset netCDF files, whether compiled with a 32-bit or 64-bit
> development environment, if the size of the C off_t type is at least 8
> bytes. The size of the off_t type for your 64-bit Windows development
> environment is surely 8 bytes, but you can test this by printing
> "sizeof(off_t)" in a small C program, and you might as well also print the
> size of size_t to answer your other question:
> 
> #include <sys/types.h>
> #include <stdio.h>
> int main(void) {
> printf("Size of off_t is %d bytes.\n", sizeof(off_t));
> printf("Size of size_t is %d bytes.\n", sizeof(size_t));
> return 0;
> }
> 
> If either of these is printed as a value less than 8, make sure you use
> whatever C compiler flags are needed for your compiler to specify a 64-bit
> development environment.
> 
> Note that there are still some limits to dataset size using the netCDF
> 64-bit offset format, as described in the Users Guide and in the FAQ on
> large file support, in particular the answers to the three questions:
> 
> Have all netCDF size limits been eliminated?
> Why are variables still limited in size?
> How can I write variables larger than 4 GiB?
> 
> http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Support10
> 
> As indicated in the answer to the second question above, the reason variable
> sizes are still limited has to do with a desire to make sure that netCDF
> 64-bit offset files files are still portable, even  to 32-bit platforms.
> 
> --Russ
> 
> >
> >
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden]
> > Sent: Monday, July 12, 2010 7:28 PM
> > To: address@hidden
> > Cc: address@hidden; address@hidden;
> > address@hidden
> > Subject: [netCDF #ZCE-849683]: clarification on large file support
> >
> > Fabio,
> >
> > A little more clarification is needed to my last reply.  I said:
> >
> > Each variable in the file cannot exceed 4GB (not 2GB), in netCDF
> > versions after 3.6.1, including the current netCDF 4.1.1. The actual
> > maximum size of a variable on a 32-bit platform is (2^32 - 4) bytes.
> > Part of the confusion is a documentation error here:
> >
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic-L
> > imitat
> > ions
> >
> > which I just discovered hasn't been updated since the size limit on a
> > single variable was changed from 2GB (2^31 - 4) to 4GB (2^32 - 4) in
> > versions since netCDF 3.6.1.
> >
> > I was confused and the original documentation is correct.  The netCDF
> > classic format limits all but the last variable to 2GB in size.  It is
> > the 64-bit offset file format that permits all variables to be 4GB in
> > size, and the last variable to be even larger.  The netCDF-4/HDF5
> > format variant has no 4GB limits on the size of any variable.  Sorry for
> the confusion!
> >
> > --Russ
> >
> >
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ZCE-849683
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> >
> >
> 
> Russ Rew                                         UCAR Unidata Program
> address@hidden                      http://www.unidata.ucar.edu
> 
> 
> 
> Ticket Details
> ===================
> Ticket ID: ZCE-849683
> Department: Support netCDF
> Priority: Normal
> Status: Closed
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: ZCE-849683
Department: Support netCDF
Priority: Normal
Status: Closed