[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #ULD-680356]: parallel netcdf: writing with increasing number of cores seems to be slower than fewer cores.

Hi,

The issue is setting up on a system with a significant number of cores; we 
simply don't have access to an HPC system for debugging this at the moment.  I 
can try to investigate it on my laptop, but even if we determine what the issue 
is, it's unlikely there is anything we can do about it at the moment since the 
Parallel I/O is handled the the hdf5 I/O interface.

We do have some improvements to parallel I/O in an upcoming release, in the 
form of a new scheduled contributed by one of our community members.  It 
appears to provide a significant increase in I/O speed.

I will follow up if we find anything in the short term; my apology for the 
delay in responding to you, but with only 2 netCDF developers and no HPC system 
available to debug on, it will take some time to find the resources for 
debugging this.  Let me go back and review the original email exchange and see 
if there's anything I can do in the short term.

Thank you, my apologies again for the delayed response,

-Ward
On Oct 20, 2018, 12:39 PM -0600, Alexandre Szenicer <address@hidden>, wrote:
> New Client Reply: parallel netcdf: writing with increasing number of cores 
> seems to be slower than fewer cores.
>
> Which part was problematic? Was it the Eigen library ? If it is, I could come 
> up with an alternative formulation only using C++ objects.
>
> Could you please provide a bit more details?
>
>
> Many thanks,
>
>
> Alex.
>
> ________________________________
> From: Unidata netCDF Support <address@hidden>
> Sent: 19 October 2018 18:36:14
> To: Alexandre Szenicer
> Cc: address@hidden; Alexandre Szenicer
> Subject: [netCDF #ULD-680356]: parallel netcdf: writing with increasing 
> number of cores seems to be slower than fewer cores.
>
> I think the problem is that we have had trouble setting up
> an environment to reproduce the problem.
>
> >
> > I've opened ticket #ULD-680356 about two months ago now, but I haven't
> > heard anything back after a first encouraging email exchange, despite
> > my inquiring for updates every two/three weeks.
> >
> > It would be very helpful to have a status on this problem, even absence
> > of progress, but not replying at all is definitely not the way to go.
> >
> > Many thanks,
> >
>
> >
> >
>
> =Dennis Heimbigner
> Unidata
>
>
> Ticket Details
> ===================
> Ticket ID: ULD-680356
> Department: Support netCDF
> Priority: Critical
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web. If you do not want to have your interactions made available in this way, 
> you must let us know in each email you send to us.
>
>
>
>
>
> Ticket Details
> ===================
> Ticket ID: ULD-680356
> Department: Support netCDF
> Priority: Critical
> Status: Open
> Link: 
> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=29707
>



Ticket Details
===================
Ticket ID: ULD-680356
Department: Support netCDF
Priority: Critical
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.