[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #UXF-258603]: NetCDF 4.2 for Windows

Hi Liz,

I've just come back from the AMS meeting in Austin, where I made a presentation
that directly addresses the problem you're describing:

  http://www.unidata.ucar.edu/presentations/Rew/ams-2013-rew-fixed.ppt
  http://www.unidata.ucar.edu/presentations/Rew/ams-2013-rew-fixed.pdf

You might want to look at it and then at the blog entries I'll be writing over 
the next week or two that have more detail about the use of nccopy for 
rechunking 
data to deal with these kinds of big data issues.

I'd like to find out if these approaches work as well for your example as they 
did
for the 38 GByte example reanalysis dataset I played with in preparing this 
talk ...

--Russ

> As a side question, do you think version 4.2 fixes what appears to
> be some sort of memory leak when doing lots of reads where I'm not grabbing
> consecutive chunks?  I'm reading in 17 classic netCDF files that each
> contain 1 meteorological variable (they are the CFSR variables like
> temperature, dewpoint, etc. from UCAR's page).  The 1.83 GB file contains 3
> dim, longitude, latitude, and time, where time is the unlimited dimension.
> I read in all dates for 1 point, i.e. 1 longitude, 1 latitude, all time
> steps using the following line:
> 
> status = nf90_get_var(ncFile,ncVarID,sData,
> &     start=(/LDim1L,LDim2L,LDim3L/),
> &     count=(/iCount1,iCount2,iCount3/))
> 
> sData is a 1-D array allocated to 745.  LDim1L is a number for the longitude
> index, e.g. 577.  LDim2L is a number of the latitude index, e.g. 160.
> LDim3L is 1.  iCount1 and iCount2 are each 1 and iCount3 is 744 (1 time step
> per hour for 0101 to 0100 of the next month).  I'm holding all 17 files open
> and then grabbing 1 variable from each file for 1 point then cycling to the
> next point and grabbing all the variables for that point, etc.  The first
> dozen or so reads of this takes 0.02 sec as part of a subroutine that also
> gets ncVarID.  After that, this read/subroutine call takes 4-7 seconds using
> the netcdf 3.5.1 library.  Using the precompiled 4.1.1 version I have, it
> takes roughly the same amount of time, but I get no reads of 0.02 sec (they
> start with the 4-7 sec versions).  I also tried using nccopy (v 4.2 from
> download below) to dump new versions of the netcdf files with NO unlimited
> dimension since I saw something online about unlimited dimensions that could
> have an effect.  Surprisingly, that made things much worse, on the order of
> 7-9 sec per read.
> 
> This might not seem so bad, but I'm trying to read 10 years (i.e.
> 120 months) at 76000 grid points.  At the optimal 4 sec that's 4 sec *17
> variables *120 months > 2 hours per point.  Even just 10000 points would
> take 2.5 years of elapsed time, which is unacceptable.
> 
> My options don't look so good, I suspect reading 1 time step for all
> points would likely improve things since that is how the file is actually
> archived, but I wouldn't be able to hold an entire file in memory since
> these arrays are 1152 x 576 x 744 (~500,000,000).  Even just 1 time step
> would be rather large and then I'd need to hold all of my selected points in
> memory as well instead of processing just 1 point at a time because I
> couldn't have more than 450 files open at once total (FORTRAN limit).  I
> suspect shrinking the netcdf files into smaller groups of grid points (say
> all longitudes but only groups of 100 latitudes or a 6 way split per file)
> might help.  Another option would be to leave netcdf format and go to
> regular direct access or even a straight PC binary format with or without a
> subset of points.
> 
> Do you have any users that have run into this similar problem?  Is
> there a sweet spot of number of open netcdf files (I tested with 6 and got
> the above results), number of points per file, and/or number of calls to
> nf90_get_var?  I'm open to any suggestions.  Please let me know if you need
> any additional information.
> 
> Regards,
> Liz
> 
> 
> -----Original Message-----
> From: Liz Orelup [mailto:address@hidden]
> Sent: Monday, January 07, 2013 11:31 AM
> To: 'address@hidden'
> Subject: RE: [netCDF #UXF-258603]: NetCDF 4.2 for Windows
> 
> Hello Ward,
> 
> Thank you for the clarification.  The web page didn't indicate that these
> pre-built libraries were C only.  I look forward to the FORTRAN release.  I
> hope it looks the same with the exe install - that was really nice!
> 
> Regards,
> Liz
> 
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: Wednesday, January 02, 2013 5:24 PM
> To: address@hidden
> Cc: address@hidden
> Subject: [netCDF #UXF-258603]: NetCDF 4.2 for Windows
> 
> Good afternoon,
> 
> > Hello,
> >
> >
> >
> > I download the 32-bit NetCDF 4.2 exe from this page:
> >
> > http://www.unidata.ucar.edu/software/netcdf/win_netcdf/
> >
> > and tried linking it as I have successfully done with an old v. 4.1.1
> > in my FORTRAN code, but I get 158 LNK2019 errors looking for NF_XXXX
> > unresolved externals.  Sneaking a text peek at the netcdf.lib file
> > (which is about half the size of the 4.1.1 one), I noticed there are
> > only nc_XXXX stubs and no NF_XXXX stubs.  I think this means that this
> > library is not FORTRAN supported and only for C users.  Do you have a
> > version for Windows FORTRAN users?
> 
> You are correct that the version you downloaded is only the C libraries;
> starting with netCDF 4.2, netCDF fortran (and C++) support is maintained in
> 
> separate libraries.  The source code for the fortran library is available
> at:
> 
> http://www.unidata.ucar.edu/downloads/netcdf/netcdf-fortran/index.jsp
> 
> Currently, we don't have a version of the fortran libraries with
> Windows support, but our goal is to introduce this support as quickly as
> possible.  Fortunately, the work which went into bringing the netCDF 4.2 C
> libraries to Windows will go a long way towards enabling Windows
> support for the fortran libraries.
> 
> I hope this information helps; I'm sorry there isn't a fortran library yet
> for
> download,  but there will be an announcement made as soon as we have a
> beta version available.
> 
> > Thanks and Happy New Year!
> >
> 
> Thank you, you too.
> 
> -Ward
> 
> > Regards,
> >
> > Liz
> > Liz Orelup
> > Meteorologist/Programmer
> > http://www.oceanweather.com <http://www.oceanweather.com/>
> > Ph: 203-661-3091
> > Email: address@hidden
> >
> >
> 
> Ticket Details
> ===================
> Ticket ID: UXF-258603
> Department: Support netCDF
> Priority: Normal
> Status: Open
> 
> 
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: UXF-258603
Department: Support netCDF
Priority: High
Status: Closed