[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[IDV #QAW-317402]: Can't get IDV 2.7u2 to image a gridded field in a netCDF file that (I think) has data stored with best practices

Hi Jim-

> "Unidata IDV Support" <address@hidden> wrote on 03/22/2010
> > 2) If that doesn't work (or if you can't upgrade), try renaming the
> > IDV resources directory:
> >
> > - rename the .unidata/idv/DefaultIdv directory under your user
> > account to DefaultIdv.bak
> > - start the IDV, load in the file and see if you get the displays to
> > show up when you click on a field.
> 
> I did this, it shows displays, and some (e.g., color-shaded plan view)
> even sort of work in that they plot color within the rectangle on the
> "Unidata IDV - Map View - One Pane".
> 
> But getting back to what I think was my original problem, I don't believe
> that IDV is using the defined LONGITUDE and LATITUDE netCDF variables
> (respectively an int and a short) to define the grid used to plot a 2d
> field.  According to the Stride tab the grid size is 909 in "x" and 505 in
> "y", which is true enough for the arrays. But as near as I can tell IDV
> does
> not make use of LONGITUDE and LATITUDE in plotting the fields.

What makes you think the IDV isn't using these values?  The netCDF library 
converts the packed lat/lon values to unpacked versions.  In the process, the 
lats go from  -139.951 to -95.0994 and the lats from 25.0 to 49.851.  For 
example, the latitude is declared as:

        short LATITUDE(lines, samples) ;
                LATITUDE:units = "" ;
                LATITUDE:missing_value = 51.f ;
                LATITUDE:valid_min = 24.99975f ;
                LATITUDE:valid_max = 49.90084f ;
                LATITUDE:scale = 0.001f ;
                LATITUDE:offset = 24.999f ;
                LATITUDE:scale_factor = 0.001f ;
                LATITUDE:add_offset = 24.999f ;

The first value is 1, so that would translate to 1*.001+24.999 which would be 
25.  For the longitudes, the declaration is:

        int LONGITUDE(lines, samples) ;
                LONGITUDE:units = "" ;
                LONGITUDE:missing_value = -94.f ;
                LONGITUDE:valid_min = -140.0005f ;
                LONGITUDE:valid_max = -95.0994f ;
                LONGITUDE:scale = 1.e-007f ;
                LONGITUDE:offset = -140.0005f ;
                LONGITUDE:scale_factor = 1.e-007f ;
                LONGITUDE:add_offset = -140.0005f ;

and the first value is 0, so that would make it 0*1E-7 + -140.0005 which would 
be -140.0005.  This is being set to missing.   Is that what you are wondering 
about or is it something different?  The last latitude is also set to NaN, but 
there, we have 24902*.001+24.999 = 49.901 which is beyond the valid_max of 
49.90084.

I got some clarification from the netCDF folks on the use of the missing and 
valid max/min values.  Here's what they replied:

<quote>
Turns out the problem is missing_value, which must be in packed units:

from http://www.unidata.ucar.edu/software/netcdf-java/v4.1/javadoc/index.html

*Implementation rules for missing data with scale/offset*

  1. Always converted to a float or double type.
  2. _FillValue and missing_value values are always in the units of the
     external (packed) data.
  3. If valid_range is the same type as scale_factor (actually the
     wider of scale_factor and add_offset) and this is wider than the
     external data, then it will be interpreted as being in the units
     of the internal (unpacked) data. Otherwise it is in the units of
     the external (packed) data.
  4. The dataType is set to float if all attributes used are float
     (scale_factor, add_offset valid_min, valid_max, valid_range,
     missing_data and _FillValue) otherwise the dataType is set to double


technically, valid_min and max should also be in packed units acording to the 
NUG, but we relaxed that in netcdf-java to accomodate some existing datasets 
that misread the manual.

also, he is using floats, and is encountering roundoff error, looking in the 
debugger i see:

valid_min = -140.00045776367188

which will also cause problems.


so:

1:  dont use missing_value or valid min/max unless you need them, eg you 
actually have missing data.
2. missing_value must be in packed units.
3. make valid_min and valid_max a little wider than you need to accomodate 
roundoff
</quote>

Also, take a look at the CF conventions about packed data:

http://cf-pcmdi.llnl.gov/documents/cf-conventions/1.4/cf-conventions.html#packed-data


> > If that works, then there is a problem with one of the preferences
> > stored in the IDV resources directory.  Have you customized the IDV
> > with a plugin
> 
> Not intentionally.
> 
> > or by changing the preferences (possibly removing the
> > displays in the Preferences->Available Displays tab)?
> 
> No, they are all set as on.

I suspect something was screwed up in the old preferences since moving them out 
of the way fixed the problem.
 
> Which architecture are you using to display my netCDF file with IDV:
> Windows, Mac, or Linux?

I use Windows, but others are using Linux.

Don Murray> 
> "Unidata IDV Support" <address@hidden> wrote on 03/22/2010
> 04:39:51 PM:
> 
> > From:
> >
> > "Unidata IDV Support" <address@hidden>
> >
> > To:
> >
> > address@hidden
> >
> > Cc:
> >
> > address@hidden, address@hidden, address@hidden
> >
> > Date:
> >
> > 03/22/2010 04:39 PM
> >
> > > Following the instructions on how to make a display with IDV I get this
> far
> > > with G11.20090602.0715.nc but no further:
> > >
> > > (Embedded image moved to file: pic15890.gif)
> >
> > I ran the file with my version of IDV 2.7u2 and got the attached
> > display.  Does the map show in your Region tab, like in mine?
> 
> The map shows in the Region tab.
> 
> > > IDV recognizes the file has 2d grids but provides no choices for making
> a
> > > display.
> >
> > It seems like something is wrong with the IDV installation.  Here
> > are a couple of things to try:
> >
> > 1) install the latest version (2.8u1) and see if you still have
> > problems.  Make sure it installs to a different directory.
> 
> I did this. It showed the map in the Region tab but no displays to choose
> from.
> 
> > 2) If that doesn't work (or if you can't upgrade), try renaming the
> > IDV resources directory:
> >
> > - rename the .unidata/idv/DefaultIdv directory under your user
> > account to DefaultIdv.bak
> > - start the IDV, load in the file and see if you get the displays to
> > show up when you click on a field.
> 
> I did this, it shows displays, and some (e.g., color-shaded plan view)
> even sort of work in that they plot color within the rectangle on the
> "Unidata IDV - Map View - One Pane".
> 
> But getting back to what I think was my original problem, I don't believe
> that IDV is using the defined LONGITUDE and LATITUDE netCDF variables
> (respectively an int and a short) to define the grid used to plot a 2d
> field.  According to the Stride tab the grid size is 909 in "x" and 505 in
> "y", which is true enough for the arrays. But as near as I can tell IDV
> does
> not make use of LONGITUDE and LATITUDE in plotting the fields.
> 
> > If that works, then there is a problem with one of the preferences
> > stored in the IDV resources directory.  Have you customized the IDV
> > with a plugin
> 
> Not intentionally.
> 
> > or by changing the preferences (possibly removing the
> > displays in the Preferences->Available Displays tab)?
> 
> No, they are all set as on.
> 
> Which architecture are you using to display my netCDF file with IDV:
> Windows, Mac, or Linux?
> 
> Jim
> 
> > > Ticket Details
> > > ===================
> > > Ticket ID: QAW-317402
> > > Department: Support IDV
> > > Priority: Critical
> > > Status: Open
> > >
> > >
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: QAW-317402
> > Department: Support IDV
> > Priority: Critical
> > Status: Open[attachment "dashboard.PNG" deleted by James F Drake/
> > Central/Aerospace/US]
> 
> 


Ticket Details
===================
Ticket ID: QAW-317402
Department: Support IDV
Priority: Critical
Status: Open