[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #KZJ-320086]: Short read are not managed?

Hi Dr. Panitz,

> with great interest I read about the problems with the short read
> described in the ticket #PDZ-683250.  But I have to admit that I did
> not understand everything written there due to "missing background".
> However, I would like to point out that I encountered the same problem
> when running a regional climate model on a CRAY XC40.  The model reads
> data from NetCDF files that are stored on a Lustre Filesystem.
> The problem is identical to that described in the ticket.
> The content of the files is not  corrupted.
> The typical size of a file is about 40 Mbyte.
> I use the following module that had been installed by CRAY staff:
> cray-netcdf/4.3.2
> Unfortunately, I cannot say how NetCDF had been installed (which options
> had been used).
> Of course, I am in contact with a member of the CRAY staff. His assumption
> is that the problems only occurs if  "there is lot of traffic on the
> filesystem".  This assumption seems to be consistent with my experience
> that the problem does not occur with every read but in an unsystematic
> manner.
> Thus, I would be very interested in any response to the ticket mentioned
> above and in any solution.  Furthermore,  I have the questions:
> - does any option exist for NetCDF installation  that could solve the
> "short read" problem?

No, sorry, there is currently no such option. Although we don't currently
test each release on a Cray platform, it's possible that an old configure
option we used to support, "--enable-ffio", might work for platforms
that support FFIO library when --disable-netcdf-4 is also specified , 
but our test platforms don't support FFIO and get compile-time 
errors for ---enable-ffio..

> - would the definition of the "bufrsize" parameter in the call of
> "nf90_open" help? If yes, which size would be appropriate,
> respectively, how can it be determined?

I don't know if it would help, but you might try a smaller size than
you're currently using. I know of no way to determine whether a
smaller size would work or what it should be other than testing with
an example that demonstrates the problem, and that may be 
impractical if it requires generating a high I/O load to see the 

We would like to have a solution for this problem, but have no
expertise in tuning netCDF on the Cray for Lustre file systems.
I see that Googling "cray lustre netcdf" turns up lots of web sites
and even an HDF5 paper on this subject: "Tuning HDF5 for Lustre 
File Systems" from The HDF Group.

> Thank your very much for any help.

Sorry I couldn't provide any useful help.


> Best regards
> Hans-Juergen Panitz
> --
> Karlsruher Institut fÃr Technologie (KIT)
> Institut fÃr Meteorologie und Klimaforschung
> Bereich TroposphÃre (IMK-TRO)
> Dr. Hans-JÃrgen Panitz
> Hermann-von-Helmholtz-Platz 1
> D-76344 Eggenstein-Leopoldshafen
> Phone: xx49-(0)721-608 22802
> Fax  : xx49-(0)721-608 24377
> E-Mail: address@hidden
> www.kit.edu
> www-fzk.imk.uni-karlsruhe.de
> KIT - UniversitÃt des Landes Baden-WÃrttemberg und
> nationales GroÃforschungszentrum in der Helmholtz-Gemeinschaft
Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu

Ticket Details
Ticket ID: KZJ-320086
Department: Support netCDF
Priority: High
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.