[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #NDY-294972]: very slow nccopy with netcdf 4.6.3



I am working on an actual fix to restore the old behavior
(which I think is the correct behavior).

> Good morning,
> 
> I was sleeping when your letter arrived. So I got the good news this
> morning and verified it when having my morning coffee. Ok, explicit
> chunksize information resolves the problem for recent netcdf versions.
> 
> I read also through the github section.
> 
> Thank you for help. Greetings,
> 
> Martin
> 
> On 5/21/19 10:51 PM, Unidata netCDF Support wrote:
> > Hi Martin,
> >
> > I don't know if you're following the GitHub thread, but it appears Dennis 
> > Heimbigner has found the issue.  At some point between 4.4.1.1 and the 
> > current versions, the default chunk sizes for netCDF/HDF5 were changed.  
> > What is happening here is that the new default chunk sizes do not lend 
> > themselves at all to the data in this file.  Dennis was able to replicate 
> > the speed found in  4.4.1.1 using the following command:
> >
> >      $ ./nccopy -k3 -d2 -c 
> > LONGITUDE481_840/360,LATITUDE81_400/320,DEPTH/1,TIME/1 wind_2008.nc 
> > ./junkd2.nc
> >
> > It appears that you may need to specify the chunksizes required when 
> > copying the netCDF files using more recent versions of netCDF; you can 
> > query these in an existing file using `ncdump -hs [filename]`.
> >
> > I hope this helps!
> >
> > -Ward
> > On May 20, 2019, 11:56 PM -0600, Martin Schmidt <address@hidden>, wrote:
> >> New Client Reply: very slow nccopy with netcdf 4.6.3
> >>
> >>
> >> Good morning,
> >>
> >> I tried to send the file, which did not work. It is to large. A subset
> >> with one time slice and one variable only is to small, the difference in
> >> execution time is not significant. I am enclosing also the time
> >> measurements with different netcdf versions again. Meanwhile I could
> >> sort out the hdf-lib as the source. I rebuild netcdf 4.4.1.1 with latest
> >> hdf5 and get sufficient results.
> >>
> >> Many thanks for your help,
> >>
> >> Martin Schmidt
> >>
> >>> module load netcdf/4.6.3-gcc7
> >>> time nccopy -k3 -d2 wind_2008.nc test.nc_4.6.3_k3_d2
> >> real    104m10.186s
> >> user    98m2.971s
> >> sys     6m3.271s
> >>
> >>> module load netcdf/4.4.1.1
> >>          Module for netcdf version 4.4.1.1 loaded
> >>> time nccopy -k3 -d2 wind_2008.nc test.nc_4.6.3_k3_d2
> >> real    0m23.760s
> >> user    0m22.563s
> >> sys     0m0.776s
> >>
> >> Same with nccopy -k4 -d2.
> >>
> >>> ncdump -k wind_2008.nc
> >> classic
> >>> ncdump -h wind_2008.nc
> >> netcdf wind_2008 {
> >> dimensions:
> >>          LONGITUDE481_840 = 360 ;
> >>          LATITUDE81_400 = 320 ;
> >>          DEPTH = 1 ;
> >>          TIME = UNLIMITED ; // (356 currently)
> >> variables:
> >>          float LONGITUDE481_840(LONGITUDE481_840) ;
> >>                  LONGITUDE481_840:long_name = "longitude" ;
> >>                  LONGITUDE481_840:units = "degrees_east" ;
> >>                  LONGITUDE481_840:axis = "X" ;
> >>                  LONGITUDE481_840:modulo = 360.f ;
> >>                  LONGITUDE481_840:point_spacing = "even" ;
> >>                  LONGITUDE481_840:standard_name = "longitude" ;
> >>          float LATITUDE81_400(LATITUDE81_400) ;
> >>                  LATITUDE81_400:long_name = "latitude" ;
> >>                  LATITUDE81_400:units = "degrees_north" ;
> >>                  LATITUDE81_400:axis = "Y" ;
> >>                  LATITUDE81_400:point_spacing = "even" ;
> >>                  LATITUDE81_400:standard_name = "latitude" ;
> >>          float DEPTH(DEPTH) ;
> >>                  DEPTH:long_name = "depth" ;
> >>                  DEPTH:units = "m" ;
> >>                  DEPTH:axis = "Z" ;
> >>                  DEPTH:positive = "up" ;
> >>                  DEPTH:point_spacing = "even" ;
> >>                  DEPTH:standard_name = "depth" ;
> >>          int TIME(TIME) ;
> >>                  TIME:long_name = "time" ;
> >>                  TIME:units = "hours since 1900-01-01 00:00:00" ;
> >>                  TIME:axis = "T" ;
> >>                  TIME:time_origin = "01-JAN-1900 00:00:00" ;
> >>                  TIME:standard_name = "time" ;
> >>          float WIND_SPEED(TIME, DEPTH, LATITUDE81_400, LONGITUDE481_840) ;
> >>                  WIND_SPEED:long_name = "wind speed" ;
> >>                  WIND_SPEED:units = "m/s" ;
> >>                  WIND_SPEED:scale_factor = 0.01 ;
> >>                  WIND_SPEED:add_offset = 0. ;
> >>                  WIND_SPEED:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  WIND_SPEED:history = "From dummy_2008" ;
> >>          float EASTWARD_WIND(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  EASTWARD_WIND:long_name = "eastward wind speed" ;
> >>                  EASTWARD_WIND:units = "m/s" ;
> >>                  EASTWARD_WIND:scale_factor = 0.01 ;
> >>                  EASTWARD_WIND:add_offset = 0. ;
> >>                  EASTWARD_WIND:positive = "meteorological convention" ;
> >>                  EASTWARD_WIND:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  EASTWARD_WIND:history = "From dummy_2008" ;
> >>          float NORTHWARD_WIND(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  NORTHWARD_WIND:long_name = "northward wind speed" ;
> >>                  NORTHWARD_WIND:units = "m/s" ;
> >>                  NORTHWARD_WIND:scale_factor = 0.01 ;
> >>                  NORTHWARD_WIND:add_offset = 0. ;
> >>                  NORTHWARD_WIND:positive = "meteorological convention" ;
> >>                  NORTHWARD_WIND:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  NORTHWARD_WIND:history = "From dummy_2008" ;
> >>          float WIND_STRESS(TIME, DEPTH, LATITUDE81_400, LONGITUDE481_840) ;
> >>                  WIND_STRESS:long_name = "wind stress" ;
> >>                  WIND_STRESS:units = "Pa" ;
> >>                  WIND_STRESS:scale_factor = 0.0001 ;
> >>                  WIND_STRESS:add_offset = 0. ;
> >>                  WIND_STRESS:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  WIND_STRESS:history = "From dummy_2008" ;
> >>          float SURFACE_DOWNWARD_EASTWARD_STRESS(TIME, DEPTH,
> >> LATITUDE81_400, LONGITUDE481_840) ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:long_name = "eastward
> >> wind stress" ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:units = "Pa" ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:scale_factor = 0.0001 ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:add_offset = 0. ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:positive =
> >> "meteorological convention" ;
> >> SURFACE_DOWNWARD_EASTWARD_STRESS:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  SURFACE_DOWNWARD_EASTWARD_STRESS:history = "From
> >> dummy_2008" ;
> >>          float SURFACE_DOWNWARD_NORTHWARD_STRESS(TIME, DEPTH,
> >> LATITUDE81_400, LONGITUDE481_840) ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:long_name =
> >> "northward wind stress" ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:units = "Pa" ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:scale_factor = 0.0001 ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:add_offset = 0. ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:positive =
> >> "meteorological convention" ;
> >> SURFACE_DOWNWARD_NORTHWARD_STRESS:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from SHORT to FLOAT" ;
> >>                  SURFACE_DOWNWARD_NORTHWARD_STRESS:history = "From
> >> dummy_2008" ;
> >>          float LAND_ICE_MASK(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  LAND_ICE_MASK:long_name = "flag - 0:ocean - 1:earth/ice" ;
> >>                  LAND_ICE_MASK:units = "1" ;
> >>                  LAND_ICE_MASK:scale_factor = 1. ;
> >>                  LAND_ICE_MASK:add_offset = 0. ;
> >>                  LAND_ICE_MASK:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from BYTE to FLOAT" ;
> >>                  LAND_ICE_MASK:history = "From dummy_2008" ;
> >>          float WIND_SPEED_RMS(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  WIND_SPEED_RMS:long_name = "wind speed root mean square" ;
> >>                  WIND_SPEED_RMS:units = "m/s" ;
> >>                  WIND_SPEED_RMS:scale_factor = 0.1 ;
> >>                  WIND_SPEED_RMS:add_offset = 0. ;
> >>                  WIND_SPEED_RMS:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from BYTE to FLOAT" ;
> >>                  WIND_SPEED_RMS:history = "From dummy_2008" ;
> >>          float EASTWARD_WIND_RMS(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  EASTWARD_WIND_RMS:long_name = "eastward wind speed root
> >> mean square" ;
> >>                  EASTWARD_WIND_RMS:units = "m/s" ;
> >>                  EASTWARD_WIND_RMS:scale_factor = 0.1 ;
> >>                  EASTWARD_WIND_RMS:add_offset = 0. ;
> >>                  EASTWARD_WIND_RMS:Ferret_Precision_Note = "This
> >> variable written using Ferret was converted from BYTE to FLOAT" ;
> >>                  EASTWARD_WIND_RMS:history = "From dummy_2008" ;
> >>          float NORTHWARD_WIND_RMS(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  NORTHWARD_WIND_RMS:long_name = "northward wind speed
> >> root mean square" ;
> >>                  NORTHWARD_WIND_RMS:units = "m/s" ;
> >>                  NORTHWARD_WIND_RMS:scale_factor = 0.1 ;
> >>                  NORTHWARD_WIND_RMS:add_offset = 0. ;
> >>                  NORTHWARD_WIND_RMS:Ferret_Precision_Note = "This
> >> variable written using Ferret was converted from BYTE to FLOAT" ;
> >>                  NORTHWARD_WIND_RMS:history = "From dummy_2008" ;
> >>          float SAMPLING_LENGTH(TIME, DEPTH, LATITUDE81_400,
> >> LONGITUDE481_840) ;
> >>                  SAMPLING_LENGTH:long_name = "sampling length" ;
> >>                  SAMPLING_LENGTH:units = "1" ;
> >>                  SAMPLING_LENGTH:scale_factor = 1. ;
> >>                  SAMPLING_LENGTH:add_offset = 0. ;
> >>                  SAMPLING_LENGTH:Ferret_Precision_Note = "This variable
> >> written using Ferret was converted from BYTE to FLOAT" ;
> >>                  SAMPLING_LENGTH:history = "From dummy_2008" ;
> >>
> >> // global attributes:
> >>                  :history = "PyFerret V7.5 (optimized) 16-May-19" ;
> >>                  :Conventions = "CF-1.6" ;
> >> }
> >>
> >> On 5/20/19 7:02 PM, Unidata netCDF Support wrote:
> >>> Can you send us that file, or at least the output
> >>> of "ncdump -h" ?
> >>>
> >>>> Dear support,
> >>>>
> >>>> I have build netcdf 4.6.3 on openSUSE and CentOS and see a very slow
> >>>> progress, very slow means 2h execution time instead of 20 s, for a
> >>>> command like this:
> >>>>
> >>>> nccopy -d2 wind_2008.nc4 wind_2008.nc4_4.6.3
> >>>>
> >>>> With nccopy from version 4.4.1.1 this task is ready in 20s.
> >>>>
> >>>> netcdf 4.6.3 is build and tested with szip 2.1.1 and hdf5 1.8.19, 1.8.20
> >>>> and 1.10.4 as well. The compiler is gcc-7, but with intel/19 and
> >>>> gcc-4.8.5 the same slow execution is observed.
> >>>>
> >>>> I am configuring with
> >>>>
> >>>> export CFLAGS="-fPIC -O2 -I$HDF_INC -I$SZIP_INC"
> >>>> export CPPFLAGS="-I$HDF_INC -I$SZIP_INC"
> >>>> export LDFLAGS="-fPIC -L$HDF_LIB -lhdf5 -lhdf5_hl -L$SZIP_LIB -lsz"
> >>>>
> >>>> ./configure --prefix=$NETCDF_ROOT \
> >>>>              --enable-netcdf-4\
> >>>>              --enable-cdf5 \
> >>>>              --enable-dap-long-tests \
> >>>>              --enable-large-file-tests \
> >>>>              --with-temp-large=/fast/mschmidt \
> >>>>              --enable-doxygen \
> >>>>              --enable-mmap \
> >>>>              --enable-jna
> >>>>
> >>>> config.status and config.log are attached.
> >>>>
> >>>> compilation fails in "include/ncconfigure.h", line 29, extern char*
> >>>> strdup(const char*); HAVE_STRDUP is defined in config.status. I do not
> >>>> understand this, but commenting this line, the compilation goes through
> >>>> and all tests are passed successfully.
> >>>>
> >>>> The library is used on a HPC linked into an ocean model and behaves well
> >>>> in this context. It is also linked into ferret and works well, except
> >>>> the "deflate" flag is simply ignored when a netcdf-file is written in
> >>>> compressed mode.
> >>>>
> >>>> I am not sure, if netcdf or hdf5 is the error source. hdf5 passes
> >>>> through all checks too. Linking with the old hdf5 that works well
> >>>> together with netcdf 4.4.1.1, does not resolve the problem. So I send
> >>>> the report to unidata first.
> >>>>
> >>>> Finally, I have also compiled the latest netcdf-c release and get the
> >>>> same behaviour.
> >>>>
> >>>>
> >>> =Dennis Heimbigner
> >>> Unidata
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: NDY-294972
> >>> Department: Support netCDF
> >>> Priority: Normal
> >>> Status: Closed
> >>> ===================
> >>> NOTE: All email exchanges with Unidata User Support are recorded in the 
> >>> Unidata inquiry tracking system and then made publicly available through 
> >>> the web. If you do not want to have your interactions made available in 
> >>> this way, you must let us know in each email you send to us.
> >>>
> >>>
> >>
> >>
> >> Ticket Details
> >> ===================
> >> Ticket ID: NDY-294972
> >> Department: Support netCDF
> >> Priority: Normal
> >> Status: Open
> >> Link: 
> >> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=30369
> >>
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: NDY-294972
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: NDY-294972
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.