[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #VUQ-958690]: Alignment of data section in NetCDF-4

Christopher,

> thank you for your answer.
> I just wanted to mention that I have found a bug in the NetCDF-4 library.
> I have raised an issue on the GitHub repository
> (see https://github.com/Unidata/netcdf-c/issues/15), but received
> no reply until today.

Thanks, I tested and committed your bug fix.

--Russ

> Zitat von Unidata netCDF Support <address@hidden>:
> 
> > Hi Christopher,
> >
> >> I am currently analysing the performance of NetCDF-4 using parallel I/O 
> >> with
> >> Lustre as underlying parallel filesystem.
> >> I was wondering, why the library does not provide the alignment
> >> feature, which aligns the data section to a particular byte boundary.
> >
> > The netCDF-3 library does permit specifying alignment, in the nc__enddef()
> > function call:
> >
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/netcdf-c.html#nc_005f_005fenddef
> >
> > but the benefits of this feature were apparently overlooked in the netCDF-4
> > implementation.  You're right, we should take advantage of alignment in
> > netCDF-4 for filesystems like Lustre.
> >
> > Also, the parallel-netCDF library based on netCDF-3 has a default
> > "nc_var_align_size"
> > alignment hint of 512 bytes, but can use other values set with a call to
> > MPI_Info_set().  The alignment setting is not stored in the file, so
> >  has to be reset
> > when open iscalled.
> >
> >> Parallel I/O performance with Lustre improves significantly when the
> >> I/O accesses are aligned to the stripe boundaries.
> >>
> >> I have added the call
> >>
> >> if( H5Pset_alignment(fapl_id, 1024*1024,1024*1024) < 0 )
> >> BAIL(NC_EHDFERR);
> >>
> >> in line 346 in nc4file.c ,
> >> recompiled the library,  and observed that the bandwith improved
> >> significantly
> >> (stripe size : 1 MiB ).
> >>
> >> Could you tell me why alignment is not provided in the NetCDF-4 library?
> >
> > Lack of resources, mostly.  I've added a Jira ticket to add this capability
> > in a future release:
> >
> >   https://bugtracking.unidata.ucar.edu/browse/NCF-289
> >
> > You've demonstrated that it should be relatively easy; we will need to do it
> > portably, so the library will work for any file system.
> >
> > Thanks for the valuable suggestion!
> >
> > --Russ
> >
> > Russ Rew                                         UCAR Unidata Program
> > address@hidden                      http://www.unidata.ucar.edu
> >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: VUQ-958690
> > Department: Support netCDF
> > Priority: High
> > Status: Closed
> >
> >
> 
> 
> 

Russ Rew                                         UCAR Unidata Program
address@hidden                      http://www.unidata.ucar.edu



Ticket Details
===================
Ticket ID: VUQ-958690
Department: Support netCDF
Priority: High
Status: Closed