[netcdf-java] Reading grib2 data occasional issues

• To: "netcdf-java@xxxxxxxxxxxxxxxx" <netcdf-java@xxxxxxxxxxxxxxxx>
• Subject: [netcdf-java] Reading grib2 data occasional issues
• From: Crawford Jeramey <Jeramey.Crawford@xxxxxxxxxxx>
• Date: Fri, 15 Nov 2019 20:25:03 +0000
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=vaisala.com; dmarc=pass action=none header.from=vaisala.com; dkim=pass header.d=vaisala.com; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=MXlZZfltaBRgZILFaU+55EI4NBw5doweK4Mz0CsW4KM=; b=gkIot3sdC5u+q8gXT1eok0b5D9jCkv15Xoiv6UheXoKYaKZCJ1eljSefbOWK9rjkrkAACal3cQy/3lMt8LDT5+ZiXzCbY6dHdUELsGuyNnBnS4feA68w9TTK73r+n1js+iAr+ZIJKhY28Vuc8rj5TynDn60iDQSgSpXV310SUgFDeQsa7fGCcbRcoREWlHB6PHPBJBEO6APrB8hOGQ2CEn6LY3RgcTmKXlT+W7Z2iuh986liNGN70u+w7IgeD9Yi7kUvvZxU/q3J9FH84LEOe6w13QlRjF3hUX5gXmz9tgGkoR87J5+W79pKxzRP37dzFlc43oaW6zREK/pqJV3v8Q==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ZiSSEAe5ZJJfQJyNwElU7n3L5vCEFJ0/yx5NossKCTklBHVixcjdle0Gn0Aap9963vpfZ30itrxJMiOSaugM0/0qI5w10tbyCdTASO6Hh6HDe4YzDoF0tHuLVq1uJdDmgeaJ3GyRCrLZs71MYbI5CQ2IeyBc6LHPvEoetGPzmb8dIabHhrLevv5NYWF8FkiJoQMNUKfJnl9Ghi1rQsqSkstws34w/rHydPTof5vhpdJjCZ38nDmGy5aQ1JjdHnM93RCd3jWgFp0QEpQHz5RW1pDkW75W3PkB3DuR+nq67PKmRfBunrWhyU3vhBY3REiDVyASqAk6jA1QZZFJ+lEu0w==
• Authentication-results: spf=none (sender IP is ) smtp.mailfrom=Jeramey.Crawford@xxxxxxxxxxx;

Hello,

I am experiencing some difficulty reading some data sets in grib2 format and 
I’m hoping someone can point out what I am doing wrong.

I have been working with several MRMS grid tables ( 
https://mrms.ncep.noaa.gov/data/2D/ ) and most of them are working just fine. 
However, I have run into some trouble with the Model_WetBulbTemp data table 
where NetCDF-Java seems to be returning spurious data.

I have written a function in Kotlin which illustrates the issue, showing the 
number of unique values in the data set and a subset of the range:

fun examineWetBulb() {
    val wetBulb = NetcdfFile.open("/tmp/MRMS_Model_WetBulbTemp.latest.grib2")
    val valueSet = mutableSetOf<Float>()
    val variable = wetBulb.findVariable("ModelWetBulbTemp_altitude_above_msl")
    val varArray = variable.read()

    for (x in 0 until variable.dimensions[2].length) {
        for (y in 0 until variable.dimensions[3].length) {
            val degC = varArray.getFloat(varArray.index.set(0, 0, x, y))
            valueSet.add(degC)
        }
    }

    println(valueSet.size)
    println(valueSet.filterIndexed { i, _ -> i % 25 == 0 })
}

When I execute this function, I see the following output:

256
[-999.0, -997.19, -998.65, -997.73, -997.48, -997.36, -997.13, -998.47, 
-998.56, -996.51, -998.86]

However, if I use an alternate implementation such as PyGRIB, I see values 
which look more likely to be correct:

>>> import pygrib
>>> wet_bulb = pygrib.open('/tmp/MRMS_Model_WetBulbTemp.latest.grib2')[1]
>>> uniq_vals = sorted(set(wet_bulb.values.flatten()))
>>> len(uniq_vals)
5168
>>> uniq_vals[0:-1:500]
[-999.0, -20.78, -15.780000000000001, -10.78, -5.78, -0.78, 4.22, 9.22, 14.22, 
19.22, 24.22]

For what it's worth, when I plot this grib2 file using Panoply, I get a similar 
result as for the NetCDF-Java case: https://i.imgur.com/xnLtxOD.png

I am using Java 8, Kotlin 1.3, and in my pom.xml file I have the following 
dependency:

        <dependency>
            <groupId>edu.ucar</groupId>
            <artifactId>grib</artifactId>
            <version>5.2.0</version>
        </dependency>
    </dependencies>

Has anyone seen this kind of problem before?

Thanks,

     --Jeramey

-- 
Jeramey Crawford
Vaisala Digital
Vaisala, Inc., Seattle, WA