Re: [thredds] Aggregated variables showing up as zeros

Hi Dave:

Can you send me a file with the problem dimension? Its probably a simple bug 
that can be fixed.

David Robertson wrote:
Hi,

I have encountered a thredds aggregation problem. I have the very latest thredds.war (3.16.30), java 1.6 and tomcat 6.0.14. The typicalDataset is set to penultimate for the aggregation. We are aggregating on the unlimited variable ocean_time. Our aggregation block in the catalog.xml file looks like this:

       <dataset name="Averages"
                ID="roms_mabgom_v4_averages"
                urlPath="roms/mabgom/v4/avg" >

<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
             <aggregation dimName="ocean_time" type="joinExisting">
<scan location="/home/om/dods-data/thredds/roms/mabgom/v4/avg/"
                      suffix=".nc" />
             </aggregation>
          </netcdf>

       </dataset>

I also looked in the aggregation cache file and all the ncoords and coordValue fields look good.

The error occurs when you ask for an ocean_time record range that does not start at the beginning of a file. For example, our nc files have 10 records each so if I ask for ocean_times 0 through 15 or 10 through 25 everything is correct and fine. However, if I ask for 1 through 15 or even 9 through 15 I get the values for records 10 through 15 then a bunch of zeros.

http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[0:1:15]

Dataset {
     Float64 ocean_time[ocean_time = 16];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[16]
4.56116478E9, 4.56120954E9, 4.5612543E9, 4.56129906E9, 4.56134382E9,
4.56138858E9, 4.56143334E9, 4.5614781E9, 4.56152286E9, 4.56156762E9,
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9


http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[1:1:15]

Dataset {
     Float64 ocean_time[ocean_time = 15];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[15]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0


http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[9:1:15]

Dataset {
     Float64 ocean_time[ocean_time = 7];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[7]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0

As you can see it's only getting records from the second file. The even stranger thing is that you get the right data (ubar in this example) using time records not starting at the beginning of a file.

http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ubar[9:1:15][0:1:0][0:1:0]

Dataset {
     Float32 ubar[ocean_time = 7][eta_u = 1][xi_u = 1];
} roms/mabgom/v4/avg;
---------------------------------------------
ubar[7][1][1]
[0][0], 0.02386481
[1][0], 0.011283709
[2][0], -0.0020005547
[3][0], -0.02091563
[4][0], -0.036607344
[5][0], -0.04679432
[6][0], -0.05165326

That data for ubar is correct. It's the last record in one file and the first 6 in another. If you tried to get the time values to line up with this data it would be wrong:

http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[9:1:15],ubar[9:1:15][0:1:0][0:1:0]

Dataset {
     Float64 ocean_time[ocean_time = 7];
     Float32 ubar[ocean_time = 7][eta_u = 1][xi_u = 1];
} roms/mabgom/v4/avg;
---------------------------------------------
ocean_time[7]
4.56161238E9, 4.56165714E9, 4.5617019E9, 4.56174666E9, 4.56179142E9,
4.56183618E9, 0.0

ubar[7][1][1]
[0][0], 0.02386481
[1][0], 0.011283709
[2][0], -0.0020005547
[3][0], -0.02091563
[4][0], -0.036607344
[5][0], -0.04679432
[6][0], -0.05165326

That first record of ocean time, you will notice, is the value for time
record 10 (first record in second file) whereas the first entry under
ubar is the value for time record 9 (last record in the first file).
This problem occurs with any aggregation set we have that has files with
more than one record in them. The only reason the problem doesn't occur
with files that have one record is because no matter what record you
start with it's at the beginning of a file.

In logs/catalina.out I get errors like:

2008-02-06T10:33:40.835 -0600 [ 1010687][ 47] ERROR - ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0001.nc

If I don't start at the beginning of a file (first time record ending in 0) and/or end at the end of a file. For Example, if I ask for records 5-15:

http://tashtego.marine.rutgers.edu:8080/thredds/dodsC/roms/mabgom/v4/avg.ascii?ocean_time[5:1:15]

produces the catalina.out errors:

2008-02-06T10:50:53.315 -0600 [ 2043167][ 712] ERROR - ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0001.nc 2008-02-06T10:50:53.316 -0600 [ 2043168][ 712] ERROR - ucar.nc2.ncml.Aggregation - readAggCoord incorrect number of coordinates dataset=/home/om/dods-data/thredds/roms/mabgom/v4/avg/mabgom_avg_0002.nc

As you can see I get one error for not starting that the beginning of a file and another error for not ending at the end of a file.

However, if I use the "NCDump Variable Data" tool from toolsUI.jar to access the same range (5-15) no errors are produced in catalina.out and the correct data values are displayed (no zeros).

Any assistance would be greatly appreciated.

Thanks,
David Robertson
_______________________________________________
thredds mailing list
thredds@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/


  • 2008 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the thredds archives: