Unidata - To provide the data services, tools, and cyberinfrastructure leadership that advance Earth system science, enhance educational opportunities, and broaden participation. Unidata
         
  advanced  
 

Re: performance degrades with filesize

Russ, your program runs in constant time for me, too.  Konrad Hinson found
the problem (in my case) is that I was calling the Python module
incorrectly that resulted in rewriting the whole scalar variable (the one
with only the unlimited dimension) every time.  The reason that I needed to
have both kinds of variables to see the problem was undoubtably that the
scalar value was needed for the slowdown, and the vector value was needed
to make the file big enough to notice it.

Basically, this fix is to change

        try:
            v[time,:] = uniform(-1, 1, tuple(list(v.shape[1:])))
        except IndexError:
            v.assignValue(uniform(-1, 1))

to 

        try:
            v[time,:] = uniform(-1, 1, tuple(list(v.shape[1:])))
        except IndexError:
            v[time] = (uniform(-1, 1))

and the python module will only write the last scalar value (which is the
correct behaviour).

I could not duplicate Konrad's result by removing the nc_inq_dimlen()
call.  I do not know what was happening with that.  Also, this problem was
pretty obscure and probably would not be seen by somebody more experienced
with NetCDF and the python interface.

One poster reminded us to set the fill value (like it says in the
netcdf documentation somewhere) for good performance, which is undoubtably
a good idea.

By the way, this is my first interaction with netcdfgroup and it was great
to get such relevant feedback so quickly.  And, I am relieved that I can just
write better code instead of hacking up the netcdf library... 8-)

John

-- 
John Galbraith                    email: john@xxxxxxxxxxxxxxx
Los Alamos National Laboratory,   home phone: (505) 662-3849
                                  work phone: (505) 665-6301

 
 
  Contact Us     Site Map     Search     Terms and Conditions     Privacy Policy     Participation Policy
 
National Science Foundation (NSF) UCAR Community Programs   Unidata is a member of the UCAR Community Programs, is managed by the University Corporation for Atmospheric Research, and is sponsored by the National Science Foundation.
P.O. Box 3000     Boulder, CO 80307-3000 USA     Tel: 303-497-8643     Fax: 303-497-8690