[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IZY-489192]: Reg : Reading netCDF files



> Hi,
> 
> I tried using the following code,
> 
> #define NTIMES 96
> #define NNODES 1181186
> float data[NTIMES][NNODES]
> nc_get_var_float(ncid, varid, &data[0][0])

Of course in C you need ";" characters terminating the last two statements.
And you should be capturing the integer returned by the nc_get_var_float 
function, which is an error code, to make sure it's NC_NOERROR (defined
in the include file you get with #include <netcdf.h> at the top of your 
C source code.  I was just writing psuedo-code for simplicity, not
whole programs, but I should have included the semi-colons.  Also, the
variables on the disk were apparently type double instead of float, but
the code above will get them as floats instead of doubles, saving half
the memory space.  If you really want to get them as doubles, call
nc_get_var_double() instead.

> Unfortunately the Console application stops working (not throwing any error
> though). I was able to find that it was failing at the line float
> data[NTIMES][NNODES]. *Is it necessary to use malloc or so to make this
> work? nc_get_var_float retrieves all the values into &data[0][0], how can i
> get/print the values from &data[0][0]?*

It's not necessary to use malloc, the line declaring data[NTIMES][NNODES]
allocates the memory local to the function it's in.  Of course, you could
call malloc instead, with a different declaration for data as a pointer to
float, but if you've got the C++ working, maybe it's better to try to use
that.

The "&data[0][0]" expression in the C function call is just an address of
the first element of a block of contiguous data in memory that will hold 
all the float values retrieved, as a 2-dimensional array.  The values are 
stored in the order data[0][0], data[0][1], ..., data[NTIMES-1][NNODES-1]. 

You can print those just as you print any values, for example:

   cout << data[time][node] << endl;

Your C++ example indicates that you'd rather use a 1-dimensional array, 
since I guess that's what a boost shared_array template gives you.

In that case, you need to use something like the alternate pseudo-code 
I provided in the comment for using a 1D array

  #define NTIMES 96
  #define NNODES 1181186
  float data[NTIMES*NNODES]; // 450 MB, so that's what, really want whole array 
at once?
  nc_get_var_float(ncid, varid, &data[0]);

> I tried using the boost shared array to obtain the values, the code works
> just fine but I do not know how to print the values. The code is as shown
> below,
> 
> 
> size_t start[] = {0,0};
> size_t end[] = {NNODES,NTIMES};
> boost::shared_array<float> temp(new float[node]);
> nc_get_vara_float (ncid, i, start, end, temp.get());

I'm assuming by "node" you mean "NTIMES*NNODES" to get a large
enough array to hold all the values, and that "i" is the
variable ID for the variable "windex".

In C++, you could print a value corresponding to time and node with:

  cout << data[NNODES * time + node] << endl ; // the data[time][node] value

Again, I don't know anything about Boost, and the C++ API is much less
used than the C API, on which it's implemented as a thin layer.  But you
can certainly use the C API from C++, as you're doing.

You originally asked if you could get all the variable values with one 
netCDF call instead of from a loop, to speed things up.  The code above
does that, but at the expense of using a *lot* of memory.  If you just want
all the time values for each node, it may be worthwhile reorganizing the
data to make time the fastest-varying dimension.  That can be done in
a number of ways, including re-chunking the data with the netCDF-4 nccopy
program, or transposing the data with the netCDF operators (NCO) ncks 
program.  

--Russ

> address@hidden> wrote:
> 
> > Hi Rekha,
> >
> > > I am trying to read a netCDF file which has the following data.
> > >
> > > [image: Inline image 1]
> >   double windx(time, node);
> >
> > (It would be easier to answer questions and archive answers for other users
> >  to find if you would paste the text into your email, rather than attaching
> >  an image of a screen display.  If you can't get the text characters any
> > other
> >  way, try looking at the short video "Optical Character Recognition (OCR)
> >  in Google Docs Tutorial", which shows one free and easy way to do it.)
> >
> > > I want to retrieve the windx values which has dimensions of time and
> > node.
> > > Here the time has 96 values and node has 1181186 values. The program runs
> > > really very slow.
> > >
> > > I have the following code, which reads 10 nodes for each timestep
> > >
> > > [image: Inline image 2]
> >
> >   if (attnames[j] == windx")
> >   {
> >       int countNodex=1;
> >       for(int incNode=0;incNode<10;incNode++)
> >       {
> >          size_t start[] = {0,incNode};
> >          size_t end[] = {95,countNodex};
> >          boost:shared_array<float> temp(new float[node]);
> >          nc_get_vara_float (ncid, i, start, end, temp.get());
> >          testing_windx.push_back(temp);
> >       }
> >       found = true;
> >   }
> >
> > > *I wanted to know whether the nc_get_vara_float reads the netCDF file
> > each
> > > time? If that is the case, then lot of computation needs to be done and
> > > hence the code can be slow. Is there any way to retrieve all the windx
> > > values from netCDF and store it in memory and then use the values from
> > the
> > > memory?*
> >
> > Yes, though I'm not familiar enough with the C++ boost library to show how
> > to
> > do it with the boost shared array, as you are trying to do above.
> >
> > In C, if you want to read all the values into a memory array declared as
> >
> >   #define NTIMES 96
> >   #define NNODES 1181186
> >   float data[NTIMES][NNODES] /* or as a 1D array, data[NTIMES*NNODES */
> >
> > you can just call nc_get_vara() once to fill up the data array, assuming
> > the 2D variable on the disk has exactly NTIMES*NNODES values:
> >
> >   nc_get_var_float(ncid, varid, &data[0][0]); /* or &data[0] for 1D array
> > */
> >
> > If there might be more values on disk, for example more times or nodes than
> > you want to handle all at once, then you need to specify start and count
> > arrays to read just the NTIMES*NNODES values, but you still need only one
> > call to nc_get_vara_float():
> >
> >   int start[] = {0, 0};
> >   int count[] = {NTIMES, NNODES};
> >     ...
> >   nc_get_vara_float(ncid, varid, start, count, &data[0][0]);
> >
> > The way you are now doing it, you read only 95 values for 95 different
> > times
> > in each loop iteration, which gets you only one value per disk access,
> > which
> > would be a fairly slow way to read the data.
> >
> > --Russ
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IZY-489192
> > Department: Support netCDF
> > Priority: Normal
> > Status: Closed
> >
> >
> 
> 
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: IZY-489192
Department: Support netCDF
Priority: Normal
Status: Closed