[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IZY-489192]: Reg : Reading netCDF files

Rekha,

> I apologize for the typing error. I do have semicolons at the end of both
> the statements. If there are no semi colons the code will not even compile.
> The problem is whenever I put the statement  float data[NTIMES][NNODES];
> the console application stops working.

This is probably due to excessive memory requirements, 450 MB, to hold
all the values in memory at once.  As I've pointed out, you can read smaller
slices into memory if you don't need them all at once.

> the console application stops working. If I comment out this statement, the
> console application just runs fine!

Great, then I'll close this ticket, since you have things working.

--Russ

> address@hidden> wrote:
> 
> > > Hi,
> > >
> > > I tried using the following code,
> > >
> > > #define NTIMES 96
> > > #define NNODES 1181186
> > > float data[NTIMES][NNODES]
> > > nc_get_var_float(ncid, varid, &data[0][0])
> >
> > Of course in C you need ";" characters terminating the last two statements.
> > And you should be capturing the integer returned by the nc_get_var_float
> > function, which is an error code, to make sure it's NC_NOERROR (defined
> > in the include file you get with #include <netcdf.h> at the top of your
> > C source code.  I was just writing psuedo-code for simplicity, not
> > whole programs, but I should have included the semi-colons.  Also, the
> > variables on the disk were apparently type double instead of float, but
> > the code above will get them as floats instead of doubles, saving half
> > the memory space.  If you really want to get them as doubles, call
> > nc_get_var_double() instead.
> >
> > > Unfortunately the Console application stops working (not throwing any
> > error
> > > though). I was able to find that it was failing at the line float
> > > data[NTIMES][NNODES]. *Is it necessary to use malloc or so to make this
> > > work? nc_get_var_float retrieves all the values into &data[0][0], how
> > can i
> > > get/print the values from &data[0][0]?*
> >
> > It's not necessary to use malloc, the line declaring data[NTIMES][NNODES]
> > allocates the memory local to the function it's in.  Of course, you could
> > call malloc instead, with a different declaration for data as a pointer to
> > float, but if you've got the C++ working, maybe it's better to try to use
> > that.
> >
> > The "&data[0][0]" expression in the C function call is just an address of
> > the first element of a block of contiguous data in memory that will hold
> > all the float values retrieved, as a 2-dimensional array.  The values are
> > stored in the order data[0][0], data[0][1], ..., data[NTIMES-1][NNODES-1].
> >
> > You can print those just as you print any values, for example:
> >
> >    cout << data[time][node] << endl;
> >
> > Your C++ example indicates that you'd rather use a 1-dimensional array,
> > since I guess that's what a boost shared_array template gives you.
> >
> > In that case, you need to use something like the alternate pseudo-code
> > I provided in the comment for using a 1D array
> >
> >   #define NTIMES 96
> >   #define NNODES 1181186
> >   float data[NTIMES*NNODES]; // 450 MB, so that's what, really want whole
> > array at once?
> >   nc_get_var_float(ncid, varid, &data[0]);
> >
> > > I tried using the boost shared array to obtain the values, the code works
> > > just fine but I do not know how to print the values. The code is as shown
> > > below,
> > >
> > >
> > > size_t start[] = {0,0};
> > > size_t end[] = {NNODES,NTIMES};
> > > boost::shared_array<float> temp(new float[node]);
> > > nc_get_vara_float (ncid, i, start, end, temp.get());
> >
> > I'm assuming by "node" you mean "NTIMES*NNODES" to get a large
> > enough array to hold all the values, and that "i" is the
> > variable ID for the variable "windex".
> >
> > In C++, you could print a value corresponding to time and node with:
> >
> >   cout << data[NNODES * time + node] << endl ; // the data[time][node]
> > value
> >
> > Again, I don't know anything about Boost, and the C++ API is much less
> > used than the C API, on which it's implemented as a thin layer.  But you
> > can certainly use the C API from C++, as you're doing.
> >
> > You originally asked if you could get all the variable values with one
> > netCDF call instead of from a loop, to speed things up.  The code above
> > does that, but at the expense of using a *lot* of memory.  If you just want
> > all the time values for each node, it may be worthwhile reorganizing the
> > data to make time the fastest-varying dimension.  That can be done in
> > a number of ways, including re-chunking the data with the netCDF-4 nccopy
> > program, or transposing the data with the netCDF operators (NCO) ncks
> > program.
> >
> > --Russ
> >
> > > address@hidden> wrote:
> > >
> > > > Hi Rekha,
> > > >
> > > > > I am trying to read a netCDF file which has the following data.
> > > > >
> > > > > [image: Inline image 1]
> > > >   double windx(time, node);
> > > >
> > > > (It would be easier to answer questions and archive answers for other
> > users
> > > >  to find if you would paste the text into your email, rather than
> > attaching
> > > >  an image of a screen display.  If you can't get the text characters
> > any
> > > > other
> > > >  way, try looking at the short video "Optical Character Recognition
> > (OCR)
> > > >  in Google Docs Tutorial", which shows one free and easy way to do it.)
> > > >
> > > > > I want to retrieve the windx values which has dimensions of time and
> > > > node.
> > > > > Here the time has 96 values and node has 1181186 values. The program
> > runs
> > > > > really very slow.
> > > > >
> > > > > I have the following code, which reads 10 nodes for each timestep
> > > > >
> > > > > [image: Inline image 2]
> > > >
> > > >   if (attnames[j] == windx")
> > > >   {
> > > >       int countNodex=1;
> > > >       for(int incNode=0;incNode<10;incNode++)
> > > >       {
> > > >          size_t start[] = {0,incNode};
> > > >          size_t end[] = {95,countNodex};
> > > >          boost:shared_array<float> temp(new float[node]);
> > > >          nc_get_vara_float (ncid, i, start, end, temp.get());
> > > >          testing_windx.push_back(temp);
> > > >       }
> > > >       found = true;
> > > >   }
> > > >
> > > > > *I wanted to know whether the nc_get_vara_float reads the netCDF file
> > > > each
> > > > > time? If that is the case, then lot of computation needs to be done
> > and
> > > > > hence the code can be slow. Is there any way to retrieve all the
> > windx
> > > > > values from netCDF and store it in memory and then use the values
> > from
> > > > the
> > > > > memory?*
> > > >
> > > > Yes, though I'm not familiar enough with the C++ boost library to show
> > how
> > > > to
> > > > do it with the boost shared array, as you are trying to do above.
> > > >
> > > > In C, if you want to read all the values into a memory array declared
> > as
> > > >
> > > >   #define NTIMES 96
> > > >   #define NNODES 1181186
> > > >   float data[NTIMES][NNODES] /* or as a 1D array, data[NTIMES*NNODES */
> > > >
> > > > you can just call nc_get_vara() once to fill up the data array,
> > assuming
> > > > the 2D variable on the disk has exactly NTIMES*NNODES values:
> > > >
> > > >   nc_get_var_float(ncid, varid, &data[0][0]); /* or &data[0] for 1D
> > array
> > > > */
> > > >
> > > > If there might be more values on disk, for example more times or nodes
> > than
> > > > you want to handle all at once, then you need to specify start and
> > count
> > > > arrays to read just the NTIMES*NNODES values, but you still need only
> > one
> > > > call to nc_get_vara_float():
> > > >
> > > >   int start[] = {0, 0};
> > > >   int count[] = {NTIMES, NNODES};
> > > >     ...
> > > >   nc_get_vara_float(ncid, varid, start, count, &data[0][0]);
> > > >
> > > > The way you are now doing it, you read only 95 values for 95 different
> > > > times
> > > > in each loop iteration, which gets you only one value per disk access,
> > > > which
> > > > would be a fairly slow way to read the data.
> > > >
> > > > --Russ
> > > >
> > > > Russ Rew                                         UCAR Unidata Program
> > > > address@hidden                      http://www.unidata.ucar.edu
> > > >
> > > >
> > > >
> > > > Ticket Details
> > > > ===================
> > > > Ticket ID: IZY-489192
> > > > Department: Support netCDF
> > > > Priority: Normal
> > > > Status: Closed
> > > >
> > > >
> > >
> > >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IZY-489192
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: IZY-489192
Department: Support netCDF
Priority: Normal
Status: Closed