[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20010410: NetCDF: poor performance writing to an NFS volume

>To: <address@hidden>
>cc: "Chris Edsall" <address@hidden>,
>cc: "Richard Rayner" <address@hidden>
>From: "Mark Hadfield" <address@hidden>
>Subject: NetCDF: poor performance writing to an NFS volume
>Organization: NIWA
>Keywords: 200104102239.f3AMddL27187

Hi Mark,

> I think the following problem is a little too technical for netcdfgroup and
> I would appreciate it if you could give me any advice about solving it.
> 
> I have a Fortran program (an ocean hydrodynamic model) that writes data to
> netCDF files. When I run it on our new Compaq Alpha machine and send output
> to an NFS volume there is a severe drop in performance.
> 
> To isolate & investigate the problem I took one of the output files and ran
> "ncgen -c" on it to generate a C program that can be run to create the file.
> (I first tried to generate a similar Fortran program with "ncgen -f", but
> the resulting program was uncompilable because of too many continuation
> lines--no  matter, the C program illustrates the problem nicely.) I modified
> the C program so I could specify the output file name on the command line
> and then ran it, directing output to local and network volumes. Here are
> elapsed times:
> 
> Local:         0.2 s
> NFS volume:    20-40 s (variable)
>
> The file in question is 1,347,576 bytes in size, by the way, and the network
> connection to the NFS volume is 100 Mbit/s ethernet. Transferring the file
> to the NFS volume via "cp" takes 1.6 s (a transfer rate of 0.8 MB/s).
> Writing via the netCDF library is 10-20 time slower than this!

I tried to duplicate the problem here, but my results seem to indicate
something wrong with the way your network or NFS server is
configured, or maybe it's very overloaded.

First, rather than use "ncgen -c" to generate a C program to run, I
just used "ncgen -b", which creates the netCDF file directly from the
CDL file by making the same calls the C program would make, with a
little extra overhead required to parse all the CDL.  I just used the
CDL you provided (from your ncdump -h output) and added made up data
for the the ocean_time variable, so there would be 721 records to
write.  The resulting file is the same size as yours (1,347,576 bytes).
(If you want to try this, I've appended the CDL file I used).

For me, creating the file on a local disk takes about 0.5 sec:

  test$ time ncgen -b -o /buddy/scratch/mark.nc mark.cdl

  real  0m0.50s
  user  0m0.10s
  sys   0m0.06s

(If I use /tmp/mark.nc as a local disk instead, I get much better
times of about 0.15 sec, but that's cheating, because /tmp is a file
system created in swap space on my workstation, so it's not really
writing to a disk.)

Then when I write to an NFS file system over a 100 Mb/sec ethernet, it
takes about 1.8 sec (varies between 1.73 sec and 2.07 sec):

  test$ time ncgen -b -o /home/russ/test/mark.nc mark.cdl

  real  0m1.83s
  user  0m0.12s
  sys   0m0.08s

or about 3.6 times as long as for a local disk.  This is the sort of NFS
performance I would expect, at least with a NFS version 3 server.
Writes to an NFS server are still slower than NFS reads, as you
noticed when you saw that copying from local disk to a remote file was
slower than copying from remote to local.  We see the same difference
here:

  test$ time cp /buddy/russ/mark.nc /home/russ/test/mark.nc

  real  0m0.72s # read local, write remote
  user  0m0.00s
  sys   0m0.02s

  test$ time cp /home/russ/test/mark.nc /buddy/russ/mark.nc

  real  0m0.19s # read remote, write local
  user  0m0.00s
  sys   0m0.02s

> Writing via the netCDF library is 10-20 time slower than this!

On our system, I see writing via netCDF is about 2.5 (1.83s/.72s)
times slower than copying, again much as expected.  I'm using a
SunOS 5.6 server (a.k.a. Solaris 2.6) and a SunOS 5.7 client
(a.k.a. Solaris 7).

Maybe your client machine (the Compaq) is running NFS version 3 and
your NFS server is still running NFS version 2.  Linux servers still
run NFS version 2, which makes them very slow NFS servers, though they
have NFS version 3 client software and work OK as clients.  We have
noticed an order of magnitude difference in speed between NFS version
3 and NFS version 2 servers.

If you have the program "nfsstat", you could see the client and server
statistics and maybe spot symptoms of a problem with your
configuration.  Or try running "netstat -i" to see if you have a lot
of I/O errors on your network.  You shouldn't have to fiddle with the
"chunksize" in nc__create; the default should work fine over NFS.

> I then changed the same program so it opens the file with nc__create instead
> of nc_create and fiddled with the chunksize argument. To cut a long story
> short I found that I could get much better network-volume performance (at
> some cost in local-volume performance) with a chunksize of 256:
> 
> Local:         0.4 s
> NFS volume:    5 s (fairly consistent)
> 
> The default chunksize allocated by the netCDF library is 8192 for local
> files and 65536 for the NFS volume. It seems to me that 256 is very small
> and something is wrong somewhere if writing the data in larger chunks than
> this "overloads the pipe".
> 
> Looking back over what I've written, I note that the transfer rate achieved
> by cp (1,347,576 bytes in 1.6 s) is pretty modest for a 100 Mbit/s
> connection, and again, the time varies quite a lot. Cp'ing the same file
> back from the NFS volume to the hard disk only takes 0.03 s or so. So I
> guess I should be looking critically at the NFS connection. But I would
> appreciate it if you could offer any comments.
> 
> Details: The machine runs Compaq Unix 5.1. The netCDF library is current
> (3.5.0) installed with no tuning or modification. The netCDF file in
> question is pretty ordinary--I have attached output from ncdump -h below. I
> won't send you a copy of the C program, since it is rather large and it was
> written by your software, but I note that it writes the data one variable at
> a time and that there are no ncsync calls anywhere that might be slowing it
> down.


--Russ

_____________________________________________________________________

Russ Rew                                         UCAR Unidata Program
address@hidden                     http://www.unidata.ucar.edu



------ mark.cdl

netcdf mark {
dimensions:
 s_rho = 50 ;
 s_w = 50 ;
 tracer = 2 ;
 station = 1 ;
 time = UNLIMITED ; // (721 currently)
variables:
 int ntimes ;
  ntimes:long_name = "number of long time-steps" ;
 int ndtfast ;
  ndtfast:long_name = "number of short time-steps" ;
 double dt ;
  dt:long_name = "size of long time-steps" ;
  dt:units = "second" ;
 double dtfast ;
  dtfast:long_name = "size of short time-steps" ;
  dtfast:units = "second" ;
 double dstart ;
  dstart:long_name = "time stamp assigned to model initilization" ;
  dstart:units = "day" ;
 int nhis ;
  nhis:long_name = "number of time-steps between history records" ;
 int nrst ;
  nrst:long_name = "number of time-steps between restart records" ;
  nrst:cycle = "only latest two records are maintained" ;
 int ntsavg ;
  ntsavg:long_name = "starting time-step for accumulation of time-averaged
fields" ;
 int navg ;
  navg:long_name = "number of time-steps between time-averaged records" ;
 int nsta ;
  nsta:long_name = "number of time-steps between stations records" ;
 double Akt_bak(tracer) ;
  Akt_bak:long_name = "background vertical mixing coefficient for tracers" ;
  Akt_bak:units = "meter2 second-1" ;
 double Akv_bak ;
  Akv_bak:long_name = "background vertical mixing coefficient for momentum"
;
  Akv_bak:units = "meter2 second-1" ;
 double rdrg ;
  rdrg:long_name = "linear drag coefficient" ;
  rdrg:units = "meter second-1" ;
 double rdrg2 ;
  rdrg2:long_name = "quadratic drag coefficient" ;
  rdrg2:units = "nondimensional" ;
 double Zo ;
  Zo:long_name = "bottom roughness" ;
  Zo:units = "meter" ;
 double Znudg ;
  Znudg:long_name = "free-surface nudging/relaxation inverse time scale" ;
  Znudg:units = "day-1" ;
 double M2nudg ;
  M2nudg:long_name = "2D momentum nudging/relaxation inverse time scale" ;
  M2nudg:units = "day-1" ;
 double M3nudg ;
  M3nudg:long_name = "3D momentum nudging/relaxation inverse time scale" ;
  M3nudg:units = "day-1" ;
 double Tnudg(tracer) ;
  Tnudg:long_name = "Tracers nudging/relaxation inverse time scale" ;
  Tnudg:units = "day-1" ;
 double rho0 ;
  rho0:long_name = "mean density used in Boussinesq approximation" ;
  rho0:units = "kilogram meter-3" ;
 double gamma2 ;
  gamma2:long_name = "Slipperiness parameter" ;
  gamma2:units = "nondimensional" ;
 char spherical ;
  spherical:long_name = "grid type logical switch" ;
  spherical:option_T = "spherical" ;
  spherical:option_F = "Cartesian" ;
 double xl ;
  xl:long_name = "domain length in the XI-direction" ;
  xl:units = "meter" ;
 double el ;
  el:long_name = "domain length in the ETA-direction" ;
  el:units = "meter" ;
 double theta_s ;
  theta_s:long_name = "S-coordinate surface control parameter" ;
  theta_s:units = "nondimensional" ;
 double theta_b ;
  theta_b:long_name = "S-coordinate bottom control parameter" ;
  theta_b:units = "nondimensional" ;
 double Tcline ;
  Tcline:long_name = "S-coordinate surface/bottom layer width" ;
  Tcline:units = "meter" ;
 double hc ;
  hc:long_name = "S-coordinate parameter, critical depth" ;
  hc:units = "meter" ;
 double sc_r(s_rho) ;
  sc_r:long_name = "S-coordinate at RHO-points" ;
  sc_r:units = "nondimensional" ;
  sc_r:valid_min = -1. ;
  sc_r:valid_max = 0. ;
  sc_r:field = "sc_r, scalar" ;
 double sc_w(s_w) ;
  sc_w:long_name = "S-coordinate at W-points" ;
  sc_w:units = "nondimensional" ;
  sc_w:valid_min = -1. ;
  sc_w:valid_max = 0. ;
  sc_w:field = "sc_w, scalar" ;
 double Cs_r(s_rho) ;
  Cs_r:long_name = "S-coordinate stretching curves at RHO-points" ;
  Cs_r:units = "nondimensional" ;
  Cs_r:valid_min = -1. ;
  Cs_r:valid_max = 0. ;
  Cs_r:field = "Cs_r, scalar" ;
 double Cs_w(s_w) ;
  Cs_w:long_name = "S-coordinate stretching curves at W-points" ;
  Cs_w:units = "nondimensional" ;
  Cs_w:valid_min = -1. ;
  Cs_w:valid_max = 0. ;
  Cs_w:field = "Cs_w, scalar" ;
 int Ipos(station) ;
  Ipos:long_name = "I-index station positions" ;
  Ipos:valid_min = 2 ;
  Ipos:valid_max = 2 ;
 int Jpos(station) ;
  Jpos:long_name = "J-index station positions" ;
  Jpos:valid_min = 2 ;
  Jpos:valid_max = 2 ;
 double h(station) ;
  h:long_name = "bathymetry at RHO-points" ;
  h:units = "meter" ;
  h:field = "bath, scalar" ;
 double x_rho(station) ;
  x_rho:long_name = "x-locations of RHO-points" ;
  x_rho:units = "meter" ;
  x_rho:field = "x_rho, scalar" ;
 double y_rho(station) ;
  y_rho:long_name = "y-locations of RHO-points" ;
  y_rho:units = "meter" ;
  y_rho:field = "y_rho, scalar" ;
 double ocean_time(time) ;
  ocean_time:long_name = "time since initialization" ;
  ocean_time:units = "second" ;
  ocean_time:field = "time, scalar, series" ;
 float zeta(time, station) ;
  zeta:long_name = "free-surface" ;
  zeta:units = "meter" ;
  zeta:field = "free-surface, scalar, series" ;
  zeta:time = "ocean_time" ;
 float ubar(time, station) ;
  ubar:long_name = "vertically integrated u-momentum component" ;
  ubar:units = "meter second-1" ;
  ubar:field = "ubar-velocity, scalar, series" ;
  ubar:time = "ocean_time" ;
 float vbar(time, station) ;
  vbar:long_name = "vertically integrated v-momentum component" ;
  vbar:units = "meter second-1" ;
  vbar:field = "vbar-velocity, scalar, series" ;
  vbar:time = "ocean_time" ;
 float u(time, station, s_rho) ;
  u:long_name = "u-momentum component" ;
  u:units = "meter second-1" ;
  u:field = "u-velocity, scalar, series" ;
  u:time = "ocean_time" ;
 float v(time, station, s_rho) ;
  v:long_name = "v-momentum component" ;
  v:units = "meter second-1" ;
  v:field = "v-velocity, scalar, series" ;
  v:time = "ocean_time" ;
 float w(time, station, s_rho) ;
  w:long_name = "vertical momentum component" ;
  w:units = "meter second-1" ;
  w:field = "w-velocity, scalar, series" ;
  w:time = "ocean_time" ;
 float temp(time, station, s_rho) ;
  temp:long_name = "potential temperature" ;
  temp:units = "Celsius" ;
  temp:field = "temperature, scalar, series" ;
  temp:time = "ocean_time" ;
 float salt(time, station, s_rho) ;
  salt:long_name = "salinity" ;
  salt:units = "PSU" ;
  salt:field = "salinity, scalar, series" ;
  salt:time = "ocean_time" ;
 float rho(time, station, s_rho) ;
  rho:long_name = "density anomaly" ;
  rho:units = "kilogram meter-3" ;
  rho:field = "density, scalar, series" ;
  rho:time = "ocean_time" ;
 float Hsbl(time, station) ;
  Hsbl:long_name = "depth of oceanic surface boundary layer" ;
  Hsbl:units = "meter" ;
  Hsbl:field = "Hsbl, scalar, series" ;
  Hsbl:time = "ocean_time" ;
 float Hbbl(time, station) ;
  Hbbl:long_name = "depth of oceanic bottom boundary layer" ;
  Hbbl:units = "meter" ;
  Hbbl:field = "Hbbl, scalar, series" ;
  Hbbl:time = "ocean_time" ;
 float AKv(time, station, s_w) ;
  AKv:long_name = "vertical viscosity coefficient" ;
  AKv:units = "meter2 second-1" ;
  AKv:field = "AKv, scalar, series" ;
  AKv:time = "ocean_time" ;
 float AKt(time, station, s_w) ;
  AKt:long_name = "temperature vertical diffusion coefficient" ;
  AKt:units = "meter2 second-1" ;
  AKt:field = "AKt, scalar, series" ;
  AKt:time = "ocean_time" ;
 float AKs(time, station, s_w) ;
  AKs:long_name = "salinity vertical diffusion coefficient" ;
  AKs:units = "meter2 second-1" ;
  AKs:field = "AKs, scalar, series" ;
  AKs:time = "ocean_time" ;
 float shflux(time, station) ;
  shflux:long_name = "surface net heat flux" ;
  shflux:units = "Watts meter-2" ;
  shflux:negative = "upward flux, cooling" ;
  shflux:positive = "downward flux, heating" ;
  shflux:field = "surface heat flux, scalar, series" ;
  shflux:time = "ocean_time" ;
 float swrad(time, station) ;
  swrad:long_name = "solar shortwave radiation flux" ;
  swrad:units = "Watts meter-2" ;
  swrad:negative = "upward flux, cooling" ;
  swrad:positive = "downward flux, heating" ;
  swrad:field = "shortwave radiation, scalar, series" ;
  swrad:time = "ocean_time" ;
 float sustr(time, station) ;
  sustr:long_name = "surface u-momentum stress" ;
  sustr:units = "Newton meter-2" ;
  sustr:field = "surface u-mometum stress, scalar, series" ;
  sustr:time = "ocean_time" ;
 float svstr(time, station) ;
  svstr:long_name = "surface v-momentum stress" ;
  svstr:units = "Newton meter-2" ;
  svstr:field = "surface v-mometum stress, scalar, series" ;
  svstr:time = "ocean_time" ;
 float bustr(time, station) ;
  bustr:long_name = "bottom u-momentum stress" ;
  bustr:units = "Newton meter-2" ;
  bustr:field = "bottom u-mometum stress, scalar, series" ;
  bustr:time = "ocean_time" ;
 float bvstr(time, station) ;
  bvstr:long_name = "bottom v-momentum stress" ;
  bvstr:units = "Newton meter-2" ;
  bvstr:field = "bottom v-mometum stress, scalar, series" ;
  bvstr:time = "ocean_time" ;

// global attributes:
  :type = "ROMS station file" ;
  :title = "ROMS 1.0 - Large et al KPP Test" ;
  :rst_file = "/tmp/roms_rst.nc" ;
  :his_file = "/tmp/roms_his.nc" ;
  :avg_file = "/tmp/roms_avg.nc" ;
  :sta_file = "/tmp/roms_sta.nc" ;
  :spos_file = "roms_sta.in" ;
  :history = "ROMS, Version 1.7.0" ;
  :CPP_options = "ANA_BSFLUX, ANA_BTFLUX, ANA_GRID, ANA_INITIAL, ANA_SMFLUX,
ANA_SRFLUX, ANA_SSFLUX, ANA_STFLUX, AVERAGES, AVERAGES_AKS, AVERAGES_AKT,
AVERAGES_AKV, DOUBLE_PRECISION, EW_PERIODIC, LMD_BKPP, LMD_CONVEC,
LMD_DDMIX, LMD_MIXING, LMD_NONLOCAL, LMD_RIMIX, LMD_SKPP, LMD_TEST,
NONLIN_EOS, NS_PERIODIC, RI_HORAVG, RI_VERAVG, SALINITY, SERIAL, SOLVE3D,
STATIONS,TS_GSCHEME, UV_ADV, UV_COR, UV_GSHEME, WJ_GRADP," ;

data:
 ocean_time = 
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36,
37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53,
54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70,
71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87,
88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103,
104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117,
118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128, 129, 130, 131,
132, 133, 134, 135, 136, 137, 138, 139, 140, 141, 142, 143, 144, 145,
146, 147, 148, 149, 150, 151, 152, 153, 154, 155, 156, 157, 158, 159,
160, 161, 162, 163, 164, 165, 166, 167, 168, 169, 170, 171, 172, 173,
174, 175, 176, 177, 178, 179, 180, 181, 182, 183, 184, 185, 186, 187,
188, 189, 190, 191, 192, 193, 194, 195, 196, 197, 198, 199, 200, 201,
202, 203, 204, 205, 206, 207, 208, 209, 210, 211, 212, 213, 214, 215,
216, 217, 218, 219, 220, 221, 222, 223, 224, 225, 226, 227, 228, 229,
230, 231, 232, 233, 234, 235, 236, 237, 238, 239, 240, 241, 242, 243,
244, 245, 246, 247, 248, 249, 250, 251, 252, 253, 254, 255, 256, 257,
258, 259, 260, 261, 262, 263, 264, 265, 266, 267, 268, 269, 270, 271,
272, 273, 274, 275, 276, 277, 278, 279, 280, 281, 282, 283, 284, 285,
286, 287, 288, 289, 290, 291, 292, 293, 294, 295, 296, 297, 298, 299,
300, 301, 302, 303, 304, 305, 306, 307, 308, 309, 310, 311, 312, 313,
314, 315, 316, 317, 318, 319, 320, 321, 322, 323, 324, 325, 326, 327,
328, 329, 330, 331, 332, 333, 334, 335, 336, 337, 338, 339, 340, 341,
342, 343, 344, 345, 346, 347, 348, 349, 350, 351, 352, 353, 354, 355,
356, 357, 358, 359, 360, 361, 362, 363, 364, 365, 366, 367, 368, 369,
370, 371, 372, 373, 374, 375, 376, 377, 378, 379, 380, 381, 382, 383,
384, 385, 386, 387, 388, 389, 390, 391, 392, 393, 394, 395, 396, 397,
398, 399, 400, 401, 402, 403, 404, 405, 406, 407, 408, 409, 410, 411,
412, 413, 414, 415, 416, 417, 418, 419, 420, 421, 422, 423, 424, 425,
426, 427, 428, 429, 430, 431, 432, 433, 434, 435, 436, 437, 438, 439,
440, 441, 442, 443, 444, 445, 446, 447, 448, 449, 450, 451, 452, 453,
454, 455, 456, 457, 458, 459, 460, 461, 462, 463, 464, 465, 466, 467,
468, 469, 470, 471, 472, 473, 474, 475, 476, 477, 478, 479, 480, 481,
482, 483, 484, 485, 486, 487, 488, 489, 490, 491, 492, 493, 494, 495,
496, 497, 498, 499, 500, 501, 502, 503, 504, 505, 506, 507, 508, 509,
510, 511, 512, 513, 514, 515, 516, 517, 518, 519, 520, 521, 522, 523,
524, 525, 526, 527, 528, 529, 530, 531, 532, 533, 534, 535, 536, 537,
538, 539, 540, 541, 542, 543, 544, 545, 546, 547, 548, 549, 550, 551,
552, 553, 554, 555, 556, 557, 558, 559, 560, 561, 562, 563, 564, 565,
566, 567, 568, 569, 570, 571, 572, 573, 574, 575, 576, 577, 578, 579,
580, 581, 582, 583, 584, 585, 586, 587, 588, 589, 590, 591, 592, 593,
594, 595, 596, 597, 598, 599, 600, 601, 602, 603, 604, 605, 606, 607,
608, 609, 610, 611, 612, 613, 614, 615, 616, 617, 618, 619, 620, 621,
622, 623, 624, 625, 626, 627, 628, 629, 630, 631, 632, 633, 634, 635,
636, 637, 638, 639, 640, 641, 642, 643, 644, 645, 646, 647, 648, 649,
650, 651, 652, 653, 654, 655, 656, 657, 658, 659, 660, 661, 662, 663,
664, 665, 666, 667, 668, 669, 670, 671, 672, 673, 674, 675, 676, 677,
678, 679, 680, 681, 682, 683, 684, 685, 686, 687, 688, 689, 690, 691,
692, 693, 694, 695, 696, 697, 698, 699, 700, 701, 702, 703, 704, 705,
706, 707, 708, 709, 710, 711, 712, 713, 714, 715, 716, 717, 718, 719,
720;

}