[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20010322: netcdf question: NCEP reanalysis: precision loss

Brent,

> To: <address@hidden>
> From: Brent A McDaniel <address@hidden>
> Subject: netcdf question
> Organization: UCAR/Unidata
> Keywords: 200103222003.f2MK3hL23783

The above message contained the following:

> Hi.  I have a netcdf question that may be simple but it's bothering me.
> I've included a plot to illustrate what I mean.  I can also give out the
> ouput of ncdump -h for both files if you'd like.
> 
> I started with ncep/ncar daily avg reanalysis data and used the Ferret
> program to do some analysis on it and then wrote a new variable to a new
> netcdf file.  Ferret writes the variable as type float (here's the line
> from ncdump -h
> float AIRANOM(TIME, LEVEL, LAT36_73, LON) ;
> .  This of course doubles the file size from the ncep file (for the same
> grid size) as the ncep files are type short.  Well, I wanted to convert
> things back to type short to save disk space.  So I used ncdump to dump
> the type float cdf file to a .cdl file, then used sed to change the float
> to short (I also removed the FillValue line which ncgen didn't seem to
> like).  Then I remade the file into cdf using ncgen.  The commands I used
> to do this are:
> 
> 
> 
> ncdump aanom.1957.nc >aanom.1957.cdl
> sed '1,50{/float/s//short/; /AIRANOM:_FillValue/d;}' aanom.1957.cdl 
> >aanom.test10.cdl
> ncgen -n aanom.test10.cdl aanom.test.cdf
> 
> And the file was, as I wanted, half the size of the float-type file
> (aanom.1857.nc).
> 
> So far so good....
> 
> But when I looked at the data I saw something unexpected:
> 
> The data in the 2 files doesn't seem to match terribly well.  It seems to
> have lost some precision in time (the flat stretches seen in the
> middle of the plot, black series) .  Is this expected?  I thought I would
> lose some accuracy on the y-axis but not the x-axis (although I didn't
> expect to lose any real accuracy...was this nieve?).
> 
> Did I do something incorrect?  Or is this just a consequence of changing
> the data type that can't be avoided?
> 
> Also, is there a better way to convert data types on the fly than what
> I've done?
> 
> Thanks for your assistance.  If this belongs on the mailing list please
> let me know and I'll subscribe to that and repost.
> 
> Cheers,
> 
> Brent
> 
> -- 
> Brent A. McDaniel
> 
> Dept of Earth and Atmospheric Sciences
> Georgia Institute of Technology
> Atlanta, Ga.  USA

I'm afraid that we don't have the resouces to would allow us to track
down this apparent discrepancy.  Due to fairly exhastive testing,
however, we're confident that the netCDF library and the ncgen and
ncdump utilities work properly.  If you can narrow the apparent problem
down to a simple example of one of these entities, then we might be able
to investigate.

One thought, however, if the original NCEP data is "packed" (i.e. uses
"scale_factor" and/or "add_offset" attributes) then I could envision the
discrepancy you're encountering.  Is the original data packed?

Regards,
Steve Emmerson   <http://www.unidata.ucar.edu>