Re: [thredds] NetCDF Subset Service

  • To: Nathan Potter <ndp@xxxxxxxxxxx>
  • Subject: Re: [thredds] NetCDF Subset Service
  • From: Christian Ward-Garrison <cwardgar@xxxxxxxx>
  • Date: Tue, 21 Jun 2016 20:08:38 -0600
Hi Nathan,

I agree with your suggestion, especially since the C++ standard library
(and perhaps C's as well?) demonstrably uses a different representation.
I've added a GitHub issue: https://github.com/Unidata/thredds/issues/569

Cheers,
Christian

On Tue, Jun 21, 2016 at 12:09 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:

>
>
> Christian,
>
> I looked through the code in Hyrax and I did some Google searching and I
> think that this is a problem that falls into gap in the IEEE-754 floating
> point specification. While I haven’t read the IEEE-754 spec, and since we
> have to pay to read it, and other sources indicate that it says pretty much
> nothing about the textual representation of the NaN, I’m going to refer to
> the free information:
>
> Wikipedia indicates that the representation is dependent on OS and
> programming language:
>   https://en.wikipedia.org/wiki/NaN#Display
>
> And this fellow:
>   http://www.working-software.com/node/34
> makes the observation (in the section titled "Special Quantities: NaN”)
> that the IEEE-754 specification doesn’t address this issue.
>
> And it does appear that C++ produces the string “nan” as opposed to “NaN”
> at least in the examples here:
>   http://en.cppreference.com/w/cpp/numeric/math/nan
>
> All of the libdap code that ingests the DAS is careful to check in a
> case-insensitive manner for the string representation of a NaN.
>
> Based on this I respectfully suggest that you consider making the
> NetCDF-Java OPeNDAP client code more lenient in this regard.
>
> Even if we patch libdap to produce the “NaN” representation (which I will)
> there are other implementations in the wild and they might produce any of
> the string representations for NaN found on the Wikipedia page.
>
>
> What do you think?
>
>
> Thanks,
>
> Nathan
>
>
>
>
>
>
>
> > On Jun 20, 2016, at 3:00 PM, Christian Ward-Garrison <cwardgar@xxxxxxxx>
> wrote:
> >
> > Hi Nathan,
> >
> > The NetCDF-Java OPeNDAP client will successfully parse a "missing_value"
> of "NaN", but not "nan". That's why you got that exception.
> >
> > I'm not sure how you ran into the lower-case variant though. For
> example, I see the following on
> http://thredds.ucar.edu/thredds/dodsC/grib/NCEP/GFS/Global_0p25deg/Best.das
> >
> >     u-component_of_wind_isobaric {
> >         String long_name "u-component of wind @ Isobaric surface";
> >         String units "m/s";
> >         String abbreviation "UGRD";
> >
> > Float32 missing_value NaN;
> >
> >         String grid_mapping "LatLon_Projection";
> >         String coordinates "reftime time isobaric lat lon ";
> >         String Grib_Variable_Id "VAR_0-2-2_L100";
> >         Int32 Grib2_Parameter 0, 2, 2;
> >         String Grib2_Parameter_Category "Momentum";
> >         String Grib2_Parameter_Name "u-component of wind";
> >         String Grib2_Level_Type "Isobaric surface";
> >         String Grib2_Generating_Process_Type "Forecast";
> >     }
> >
> >
> > On Mon, Jun 20, 2016 at 12:11 PM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> >
> >
> > Rich,
> >
> >
> > That’s different software using a different interface to access the
> data. While the TDS uses “ncWMS” as a library to provide WMS services for
> data it does not, as far as I know utilize:
> >
> > a) ncWMS2 which is the latest WMS software from the Reading Science
> Center
> >
> > b) The Java-NetCDF libraries ability to read DAP datasets as a mechanism
> for accessing data for it’s “captive” WMS.
> >
> >
> > So that’s good that it works in TDS, but really my question is about
> Java-NetCDF failing to correctly identify/process missing_value attributes
> whose value is “nan”, which is the failure mode that ncWMS2 is experiencing.
> >
> > Interestingly, it doesn’t appear to be a problem when I direct ncWMS2 to
> directly access the netcdf subset file. It’s only a problem when ncWMS2 is
> accessing the dataset via DAP2 (which is done through the Java-NetCDF
> library)
> >
> > I;ve cc’d Unidata support TDS-users. Maybe someone there has had this
> experience or knows a work around.
> >
> > Thanks for the help Rich!
> >
> > Nathan
> >
> >
> >
> >
> > > On Jun 20, 2016, at 10:31 AM, Signell, Richard <rsignell@xxxxxxxx>
> wrote:
> > >
> > > This WMS request seems to work okay:
> > >
> http://thredds.ucar.edu/thredds/wms/grib/NCEP/GFS/Global_0p25deg/Best?LAYERS=u-component_of_wind_isobaric&ELEVATION=100&TIME=2016-06-20T00%3A00%3A00.000Z&TRANSPARENT=true&STYLES=boxfill%2Frainbow&COLORSCALERANGE=-52.3%2C160&NUMCOLORBANDS=20&LOGSCALE=false&SERVICE=WMS&VERSION=1.1.1&REQUEST=GetMap&EXCEPTIONS=application%2Fvnd.ogc.se_inimage&FORMAT=image%2Fpng&SRS=EPSG%3A4326&BBOX=-180,-90,180,90&WIDTH=256&HEIGHT=256
> > >
> > > On Mon, Jun 20, 2016 at 12:34 PM, Nathan Potter <ndp@xxxxxxxxxxx>
> wrote:
> > >> I got that datset-subset but I can’t get ncWMS to load it because:
> > >>
> > >> 2016-06-20 09:22:03 ERROR NetcdfDataset:1043 - Error
> openDodsByReflection:
> > >> java.lang.IllegalArgumentException: Illegal Numeric Value for
> Attribute
> > >> Value for missing_value
> > >> at ucar.nc2.dods.DODSAttribute.<init>(DODSAttribute.java:114)
> > >> at ucar.nc2.dods.DodsV.addAttribute(DodsV.java:441)
> > >> at ucar.nc2.dods.DodsV.addAttributeTable(DodsV.java:432)
> > >> at ucar.nc2.dods.DodsV.parseDAS(DodsV.java:407)
> > >> at ucar.nc2.dods.DODSNetcdfFile.<init>(DODSNetcdfFile.java:296)
> > >> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> > >> at
> > >>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > >> at
> > >>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > >> at java.lang.reflect.Constructor.newInstance(Constructor.java:422)
> > >> at
> > >>
> ucar.nc2.dataset.NetcdfDataset.openDodsByReflection(NetcdfDataset.java:1040)
> > >>
> > >> Which looks like a bug down inside Java-NetCDF.
> > >>
> > >> At least to me it looks like a bug, from the DAS:
> > >>
> > >>    u-component_of_wind_isobaric {
> > >>        String long_name "u-component of wind @ Isobaric surface";
> > >>        String units "m/s";
> > >>        String abbreviation "UGRD";
> > >>        Float32 missing_value nan;
> > >>        String grid_mapping "LatLon_Projection";
> > >>        String coordinates "reftime time isobaric lat lon lat lon";
> > >>        String Grib_Variable_Id "VAR_0-2-2_L100";
> > >>        Int32 Grib2_Parameter 0, 2, 2;
> > >>        String Grib2_Parameter_Category "Momentum";
> > >>        String Grib2_Parameter_Name "u-component of wind";
> > >>        String Grib2_Level_Type "Isobaric surface";
> > >>        String Grib2_Generating_Process_Type "Forecast";
> > >>    }
> > >>
> > >>
> > >>
> > >> Do you know if this is a common problem with an easy workaround, or
> should I
> > >> badger Unidata with a bug report?
> > >>
> > >> N
> > >>
> > >>
> > >>
> > >> On Jun 17, 2016, at 9:16 AM, Nathan Potter <ndp@xxxxxxxxxxx> wrote:
> > >>
> > >> Rich,
> > >>
> > >> Yeah, I do want all that. We are working on WCS for 3D+time (4
> independent
> > >> vars) and I was working to get some real test datasets.
> > >>
> > >>
> > >> N
> > >>
> > >>
> > >> On Jun 17, 2016, at 7:41 AM, Signell, Richard <rsignell@xxxxxxxx>
> wrote:
> > >>
> > >> Nathan,
> > >> I took your same request, but changed the horizontal stride from 1 to
> > >> 100 (horizStride=100)
> > >>
> > >>
> http://thredds.ucar.edu/thredds/ncss/grib/NCEP/GFS/Global_0p25deg/best?var=u-component_of_wind_isobaric&var=v-component_of_wind_isobaric&horizStride=100&time_start=2016-06-13T00%3A00%3A00Z&time_end=2016-07-02T06%3A00%3A00Z&timeStride=1&vertCoord=&addLatLon=true&accept=netcdf4
> > >>
> > >> It took 12 minutes, but successfully completed the request, delivering
> > >> a 2MB netcdf4 file which I've attached and placed here:
> > >>
> > >>
> http://geoport-dev.whoi.edu/thredds/catalog/usgs/data2/rsignell/test/catalog.html?dataset=usgs/data2/rsignell/test/best.nc
> > >>
> > >> Checking out the DDS at
> > >>
> http://geoport-dev.whoi.edu/thredds/dodsC/usgs/data2/rsignell/test/best.nc.dds
> > >> I see that you are asking for 117 time steps, and there are 31
> > >> vertical levels in these variables.
> > >>
> > >> Did you really want all that?
> > >>
> > >> Your original request (with horizStride=1) would have been a request
> > >> for 100x100=10,000 times more data, or roughly 20GB of data.   Perhaps
> > >> that's why it crapped out?
> > >>
> > >> But I agree the original error message you received was not very
> > >> informative.
> > >>
> > >> -Rich
> > >>
> > >>
> > >>
> > >> On Thu, Jun 16, 2016 at 1:18 PM, Nathan Potter <ndp@xxxxxxxxxxx>
> wrote:
> > >>
> > >>
> > >> I used this form:
> > >>
> > >>
> http://thredds.ucar.edu/thredds/ncss/grib/NCEP/GFS/Global_0p25deg/best/dataset.html
> > >>
> > >> To construct this NetCDF Subset Service request:
> > >>
> > >>
> http://thredds.ucar.edu/thredds/ncss/grib/NCEP/GFS/Global_0p25deg/best?var=u-component_of_wind_isobaric&var=v-component_of_wind_isobaric&horizStride=1&time_start=2016-06-13T00%3A00%3A00Z&time_end=2016-07-02T06%3A00%3A00Z&timeStride=1&vertCoord=&addLatLon=true&accept=netcdf4
> > >>
> > >> Which returns the following remarkably cryptic message:
> > >>
> > >> first (-66) must be >= 0
> > >>
> > >> Does anybody have some thoughts as to what I might change to make the
> > >> request successful, or to get a more understandable message?
> > >>
> > >>
> > >> Thanks,
> > >>
> > >> Nathan
> > >>
> > >>
> > >> = = =
> > >> Nathan Potter                        ndp at opendap.org
> > >> OPeNDAP, Inc.                        +1.541.231.3317
> > >>
> > >> _______________________________________________
> > >> NOTE: All exchanges posted to Unidata maintained email lists are
> > >> recorded in the Unidata inquiry tracking system and made publicly
> > >> available through the web.  Users who post to any of the lists we
> > >> maintain are reminded to remove any personal information that they
> > >> do not want to be made public.
> > >>
> > >>
> > >> thredds mailing list
> > >> thredds@xxxxxxxxxxxxxxxx
> > >> For list information or to unsubscribe,  visit:
> > >> http://www.unidata.ucar.edu/mailing_lists/
> > >>
> > >>
> > >>
> > >>
> > >> --
> > >> Dr. Richard P. Signell   (508) 457-2229
> > >> USGS, 384 Woods Hole Rd.
> > >> Woods Hole, MA 02543-1598
> > >> <best.nc4>
> > >>
> > >>
> > >> = = =
> > >> Nathan Potter                        ndp at opendap.org
> > >> OPeNDAP, Inc.                        +1.541.231.3317
> > >>
> > >>
> > >> = = =
> > >> Nathan Potter                        ndp at opendap.org
> > >> OPeNDAP, Inc.                        +1.541.231.3317
> > >>
> > >
> > >
> > >
> > > --
> > > Dr. Richard P. Signell   (508) 457-2229
> > > USGS, 384 Woods Hole Rd.
> > > Woods Hole, MA 02543-1598
> >
> > = = =
> > Nathan Potter                        ndp at opendap.org
> > OPeNDAP, Inc.                        +1.541.231.3317
> >
> > _______________________________________________
> > NOTE: All exchanges posted to Unidata maintained email lists are
> > recorded in the Unidata inquiry tracking system and made publicly
> > available through the web.  Users who post to any of the lists we
> > maintain are reminded to remove any personal information that they
> > do not want to be made public.
> >
> >
> > thredds mailing list
> > thredds@xxxxxxxxxxxxxxxx
> > For list information or to unsubscribe,  visit:
> http://www.unidata.ucar.edu/mailing_lists/
> >
>
> = = =
> Nathan Potter                        ndp at opendap.org
> OPeNDAP, Inc.                        +1.541.231.3317
>
>
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: