[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #WVW-312002]: serial netcdf unexplained behaviour

Hello,

I'm a bit confused by this example; you reference serial netCDF, but clearly 
the netCDF library you are using has been built with parallel I/O support? Am I 
reading this correctly?

I suspect the overhead you are seeing is that all the metadata for the file is 
being written with the first write, in addition to the data.  There is overhead 
that goes into creating a new file, in addition to simply writing the data.

Try this; create a test program that only creates the file and defines the 
structure.  Then, create a second test program that only opens the file and 
writes the data.  Does the delay persist in the second program, during the 
first loop iteration? Or is the delay observed in the former program, when the 
file is created?

-Ward


> Hello,
> 
> I am writing to you because I am currently encountering issues with
> serial netcdf. Indeed, I was using parallel netcdf but I'm also having
> some issues (currently the subject of another ticket with the support),
> so I decided to write a serial version of my code. However, I'm still
> running into some problems.
> 
> I reproduced my problem in a toy example that you should be easily able
> to run: you just need the Eigen library. I attached the toy example to
> this email, just modify the paths in the makefile, and you should be
> good to go.
> 
> So what happens is: I'm trying to write a 5 dimensional array (that I
> split between the cores) in serial. The trick is, I need to loop over
> the first dimension, and write 4D arrays chunk by chunk (why? this is
> the subject of another ticket I have submitted, where if I write the
> whole array at once it produces garbage values in the netcdf file). So
> the weird thing is, the first iteration of the "write loop", or if you
> wish the first call to write, takes much much longer (about 10x) than the
> subsequent iterations. You can imagine for larger files it significantly
> slows down my code. What's also interesting is that this behavior does
> not happen in parallel netcdf, all w rites take about the same time.
> 
> To test this, please just run the example, and you should see the write
> counter appearing on your terminal. Please let me know if anything
> is unclear.
> 
> Many thanks for your assistance,
> 
> Alexandre Szenicer.
> 
> 
> Hello,
> 
> I am writing to you because I am currently encountering issues with
> serial netcdf. Indeed, I was using parallel netcdf but I'm also having
> some issues (currently the subject of another ticket with the support),
> so I decided to write a serial version of my code. However, I'm still
> running into some problems.
> 
> I reproduced my problem in a toy example that you should be easily able
> to run: you just need the Eigen library. I attached the toy example to
> this email, just modify the paths in the makefile, and you should be
> good to go.
> 
> So what happens is: I'm trying to write a 5 dimensional array (that I
> split between the cores) in serial. The trick is, I need to loop over
> the first dimension, and write 4D arrays chunk by chunk (why? this is
> the subject of another ticket I have submitted, where if I write the
> whole array at once it produces garbage values in the netcdf file). So
> the weird thing is, the first iteration of the "write loop", or if you
> wish the first call to write, takes much much longer (about 10x) than the
> subsequent iterations. You can imagine for larger files it significantly
> slows down my code. What's also interesting is that this behavior does
> not happen in parallel netcdf, all w rites take about the same time.
> 
> To test this, please just run the example, and you should see the write
> counter appearing on your terminal. Please let me know if anything
> is unclear.
> 
> Many thanks for your assistance,
> 
> Alexandre Szenicer.
> 
> 



Ticket Details
===================
Ticket ID: WVW-312002
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.