> I'm quite sorry,
> Actually I'm writing much more than what I stated in my previous email
>
> I'm trying to dump on disk about 9GB. But It does not work and provides me
> the error I already said.
...
> I'm still trying to write a large file on a 64bit platform.
>
> I fixed the variables type and now I get 8 bytes for both size_t and off_t
> system variables.
>
> ... I get an error: 22
> ("invalis argument") from the nc_endded function.
>
> What is the cause?
The netCDF error code for "Invalid Argument" is (-36), so the error
you are getting of 22 is from the operating system, in /usr/include/errno.h:
#define EINVAL 22 /* Invalid argument */
This may still be a netCDF library problem, but there are other reasons
related to your file system or operating system that may be the cause. See
the answer to the FAQ "Why do I get an error message when I try to create a
file larger than 2 GiB with the new library?":
http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Support12
Can you create large files with any other program on your system? For example,
do you get an error from the operating system trying to write a 5GB file with a
program that doesn't use the netCDF library, such as the following little C
program? If so, try compiling it with -D_FILE_OFFSET_BITS=64 flag. If that
still doesn't work, you'll need to configure your file system for large file
support.
--Russ
/*
* See if we can write a 5GB file, to make sure large file system
* supported. Note: may need to compile this with
* -D_FILE_OFFSET_BITS=64 flag on 32-bit systems to get an 8-byte
* off_t type.
*/
#include <stdlib.h>
#include <stdio.h>
#include <string.h>
#include <errno.h>
extern int errno;
#define MILLION 1000000
int
main() {
int i, j, k;
float data[MILLION];
size_t written;
FILE *f;
f = fopen("bigfile", "w");
if(!f) {
printf("error opening file: %s\n", strerror(errno));
return errno;
}
for(i=0; i < 1250; i++) {
written = fwrite(data, sizeof(float), MILLION, f);
if (written < MILLION) {
printf("error writing, errno=%d, %s\n", errno, strerror(errno));
return errno;
}
}
return 0;
}
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden
> Sent: Friday, July 23, 2010 6:26 PM
> To: address@hidden
> Cc: address@hidden; address@hidden;
> address@hidden; address@hidden
> Subject: [netCDF #ZCE-849683]: clarification on large file support
>
> > I did as you suggested me.
> >
> > But I get the following outcome:
> >
> > Size of Size_t = 8 ok
> > Size of Off_t = 4 !unbelievable
> >
> > How does it come?
> > We expect both of quantities to be equal to 8 bytes on a 64 bit
> > platform, don't they?
>
> I'm surprised that the default size of off_t is only 4, but maybe you need
> to set a C macro to get an 8-byte off_t. Try compiling and running this C
> program:
>
> #define _FILE_OFFSET_BITS 64
> #include <sys/types.h>
> #include <stdio.h>
> int main(void) {
> printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of
> size_t is %d bytes.\n", sizeof(size_t)); return 0; }
>
> If that shows sizeof(off_t) is now 8 bytes, then you can rebuild netCDF
> setting the CFLAGS environment variable, for example
>
> CFLAGS='-D_FILE_OFFSET_BITS=64'
>
> > Maybe some particular preprocessor have to be defined before compiling
> > the netcdf lib?
> > I can find the following preprocessors in my solution:
> > _WINDOWS;_USRDLL;NETCDF_EXPORTS;VERSION=3.6.1-beta1;DLL_EXPORT;VISUAL_
> > CPLUSP
> > LUS;_FILE_OFFSET_BITS=64
> >
> > I removed the NETCDF_DLL preprocessor since I need to create a static
> > library rather then a dynamic one.
> >
> > Can you give me some more suggestions to get the library work properly
> > with large file support?
>
> I wish I knew more about the Windows compilation environment for 64-bit
> support. We may get these problems solved in the next six months with our
> planned port to Windows that was recently announced on the netcdfgroup
> mailing list:
>
>
> http://www.unidata.ucar.edu/mailing_lists/archives/netcdfgroup/2010/msg00257
> .html
>
> --Russ
>
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden
> > Sent: Monday, July 19, 2010 6:15 PM
> > To: address@hidden
> > Cc: address@hidden; address@hidden;
> > address@hidden; address@hidden
> > Subject: [netCDF #ZCE-849683]: clarification on large file support
> >
> > Fabio,
> >
> > > Please, I need just one more clarification.
> > > I would like to use the netcdf library on windows 64 bit platform.
> > > In order to activate very large file support, is it sufficient to
> > > compile the library specifying a 64bit environment and activate the
> > > 64bit file offset. Doing so, does the size_t type automatically
> > > switch to a 64 bit unsigned integer variable in such a way not to
> > > limit memory and thus data set allocation. I am using your code and
> > > Visual studio 2008 and I'm specifing a 64 bit target environment.
> >
> > With a 64-bit development environment, there is no need to "activate
> > the 64bit file offset", and this depends on the off_t type, not size_t.
> >
> > All versions of netCDF since 3.6.0 include support for reading and
> > writing 64-bit offset netCDF files, whether compiled with a 32-bit or
> > 64-bit development environment, if the size of the C off_t type is at
> > least 8 bytes. The size of the off_t type for your 64-bit Windows
> > development environment is surely 8 bytes, but you can test this by
> > printing "sizeof(off_t)" in a small C program, and you might as well
> > also print the size of size_t to answer your other question:
> >
> > #include <sys/types.h>
> > #include <stdio.h>
> > int main(void) {
> > printf("Size of off_t is %d bytes.\n", sizeof(off_t)); printf("Size of
> > size_t is %d bytes.\n", sizeof(size_t)); return 0; }
> >
> > If either of these is printed as a value less than 8, make sure you
> > use whatever C compiler flags are needed for your compiler to specify
> > a 64-bit development environment.
> >
> > Note that there are still some limits to dataset size using the netCDF
> > 64-bit offset format, as described in the Users Guide and in the FAQ
> > on large file support, in particular the answers to the three questions:
> >
> > Have all netCDF size limits been eliminated?
> > Why are variables still limited in size?
> > How can I write variables larger than 4 GiB?
> >
> > http://www.unidata.ucar.edu/netcdf/docs/faq.html#Large%20File%20Suppor
> > t10
> >
> > As indicated in the answer to the second question above, the reason
> > variable sizes are still limited has to do with a desire to make sure
> > that netCDF 64-bit offset files files are still portable, even to 32-bit
> platforms.
> >
> > --Russ
> >
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata netCDF Support
> > > [mailto:address@hidden
> > > Sent: Monday, July 12, 2010 7:28 PM
> > > To: address@hidden
> > > Cc: address@hidden; address@hidden;
> > > address@hidden
> > > Subject: [netCDF #ZCE-849683]: clarification on large file support
> > >
> > > Fabio,
> > >
> > > A little more clarification is needed to my last reply. I said:
> > >
> > > Each variable in the file cannot exceed 4GB (not 2GB), in netCDF
> > > versions after 3.6.1, including the current netCDF 4.1.1. The actual
> > > maximum size of a variable on a 32-bit platform is (2^32 - 4) bytes.
> > > Part of the confusion is a documentation error here:
> > >
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html#Classic
> > > -L
> > > imitat
> > > ions
> > >
> > > which I just discovered hasn't been updated since the size limit on
> > > a single variable was changed from 2GB (2^31 - 4) to 4GB (2^32 - 4)
> > > in versions since netCDF 3.6.1.
> > >
> > > I was confused and the original documentation is correct. The
> > > netCDF classic format limits all but the last variable to 2GB in
> > > size. It is the 64-bit offset file format that permits all
> > > variables to be 4GB in size, and the last variable to be even
> > > larger. The netCDF-4/HDF5 format variant has no 4GB limits on the
> > > size of any variable. Sorry for
> > the confusion!
> > >
> > > --Russ
> > >
> > >
> > >
> > > Russ Rew UCAR Unidata Program
> > > address@hidden http://www.unidata.ucar.edu
> > >
> > >
> > >
> > > Ticket Details
> > > ===================
> > > Ticket ID: ZCE-849683
> > > Department: Support netCDF
> > > Priority: Normal
> > > Status: Closed
> > >
> > >
> > >
> > >
> >
> > Russ Rew UCAR Unidata Program
> > address@hidden http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: ZCE-849683
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
>
> Russ Rew UCAR Unidata Program
> address@hidden http://www.unidata.ucar.edu
>
>
>
> Ticket Details
> ===================
> Ticket ID: ZCE-849683
> Department: Support netCDF
> Priority: Normal
> Status: Closed
>
>
Russ Rew UCAR Unidata Program
address@hidden http://www.unidata.ucar.edu
Ticket Details
===================
Ticket ID: ZCE-849683
Department: Support netCDF
Priority: Normal
Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.