NetCDF-4 AR-4 Performance Data With Horizonatal and Time Series Reversed

Russ, Dennis and I discussed some of the chunking results yesterday. We were concerned that the horizontal reads were causing all the data to be preloaded into cache for the subsequent time series read. So I swapped the order - now the time series read is done first.

Here's some results. The first line are the results for reading a classic netCDF file.

cs[0] cs[1] cs[2] cache(MB) deflate shuffle read_hor(us)  read_time_ser(us)
0     0     0     0         0      0        247           5908
256   64    128   4         0      0        241           2039
256   64    128   32        0      0        168           31384
256   64    128   128       0      0        140           17096
256   64    256   4         0      0        93            2548
256   64    256   32        0      0        136           55722
256   64    256   128       0      0        106           26892
256   128   128   4         0      0        216           2035
256   128   128   32        0      0        152           55488
256   128   128   128       0      0        121           26698
256   128   256   4         0      0        79            2392
256   128   256   32        0      0        188           191120
256   128   256   128       0      0        136           186396
1024  64    128   4         0      0        236           1945
1024  64    128   32        0      0        108356        53812
1024  64    128   128       0      0        220           19551
1024  64    256   4         0      0        89            1930
1024  64    256   32        0      0        89            1864
1024  64    256   128       0      0        209           40942
1024  128   128   4         0      0        222           2065
1024  128   128   32        0      0        220           1833
1024  128   128   128       0      0        227           41183
1024  128   256   4         0      0        77            1894
1024  128   256   32        0      0        76            1839
1024  128   256   128       0      0        199           207533
1560  64    128   4         0      0        234           1885
1560  64    128   32        0      0        233           1850
1560  64    128   128       0      0        161596        14921
1560  64    256   4         0      0        88            1969
1560  64    256   32        0      0        87            1929
1560  64    256   128       0      0        160939        30848
1560  128   128   4         0      0        218           1924
1560  128   128   32        0      0        218           1875
1560  128   128   128       0      0        161316        30876
1560  128   256   4         0      0        77            1857
1560  128   256   32        0      0        76            1797
1560  128   256   128       0      0        76            1796

Again, there are many chunk size selections which beat the classic netCDF file performance.

Comments:

Post a Comment:
Comments are closed for this entry.
Unidata Developer's Blog
A weblog about software development by Unidata developers*
Unidata Developer's Blog
A weblog about software development by Unidata developers*

Welcome

FAQs

News@Unidata blog

Take a poll!

What if we had an ongoing user poll in here?

Browse By Topic
Browse by Topic
« December 2019
SunMonTueWedThuFriSat
1
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
    
       
Today