[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Support #JWA-773791]: building best estimate aggregations using ncml



> Hi all,
> 
> I have a few questions about Thredds aggregations.
> 
> My first goal (1) is to make a best estimate view of forecast model runs
> updated every days (in this example because it could be every 6 hours, ..)
> These runs have more than 24 hours forecast so I'd like to aggregate only
> the first 24 time steps of each files.
> 
> Moreover (2), I need to build that kind of dataset with another aggregation
> of type union to merge netcdf files because different parameters (variables)
> are separated in many files (but for sure, share the same time axis values)
> 
> I think (after having discussed about it here quite long time ago..) 

where is "here" ?

it is
> not possible to specify an offset for selecting for example
> the T+24 => T+48 part of a file, however it seems to be possible to select
> the T => T+24 part of files using the ncoords attribute of the netcdf
> elements.
> That was the way I found to build the best estimate views using thredds 3.16
> 
> In these examples, the data files contains 73 time coords, hourly. So
> wrfBRE_d03_temp_2009-01-01T00Z.nc contain data from 2009-01-01T00Z to
> 2009-01-04T00Z
> Taking the 24 first time coords of this file would give a period from
> 2009-01-01T00Z to 2009-01-01T23Z
> 
> 1) JoinExisting aggregation
> ===========================
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> 
> Using thredds 3.16 with these configurations, I use the Dataset Query Form,
> select only the time variable with the default proposed slice:  0:1:144, but
> when
> I submit the form, I get the error
> Error {
> code = -1;
> message = "NcSDArray ArrayIndexOutOfBoundsException=null";
> };
> 
> With the slice 0:1:143, I get the time values correctly
> 
> Any idea why I cannot get the last time coord ?
> 
> Using thredds 4.0.26, I get the time values correctly in both cases

bug was fixed in 4.0

> 
> 
> 2) JoinExisting + Union aggregations
> ====================================
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-01T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-02T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-03T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf>
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> </aggregation>
> </netcdf>
> 
> I still have the last time coord problem using thredds 3.16, but the N-1
> available time values are still correct
> 
> Using thredds 4.0.26, I get the 144 time values but these are strangely
> corrupted, I mean it's like the 24 time values of the first
> union aggregation (time values of wrfBRE_d03_temp_2009-01-01T00Z.nc) are
> reproduced four times: I get the same values for the four
> union aggregations !
> This give me the following time evolution
> 2009-01-01T00Z => 2009-01-04T00Z => 2009-01-01T00Z => 2009-01-04T00Z =>
> 2009-01-01T00Z => 2009-01-04T00Z => 2009-01-01T00Z => 2009-01-04T00Z
> 
> 
> Now adding the ncoords attribute to the union's parent netcdf element like
> this:
> 
> <netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2";>
> <aggregation dimName="time" type="joinExisting">
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-01T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-01T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-02T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-02T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="24">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-03T00Z.nc"
> ncoords="24"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-03T00Z.nc"
> ncoords="24"/>
> </aggregation>
> </netcdf>
> <netcdf ncoords="73">
> <aggregation dimName="time" type="union">
> <netcdf
> location="file:///data/runs/wrfBRE_d03_temp_2009-01-04T00Z.nc"
> ncoords="73"/>
> <netcdf
> location="file:///data/runs/wrfBRE_d03_wind_2009-01-04T00Z.nc"
> ncoords="73"/>
> </aggregation>
> </netcdf>
> </aggregation>
> </netcdf>
> 
> Then I get quite the same behavior but the resulting period is varying
> between each call (submit)
> starting at 2009-01-01T00Z or 2009-01-02T00Z, or 2009-01-03T00Z or
> 2009-01-04T00Z
> and getting for example when starting at 2009-01-01T00Z, the following
> evolution:
> 2009-01-01T00Z => 2009-01-02T00Z => 2009-01-01T00Z => 2009-01-02T00Z =>
> 2009-01-01T00Z => 2009-01-02T00Z => 2009-01-01T00Z => 2009-01-04T00Z
> 
> (when the result starts from 2009-01-02T00Z:
> 2009-01-02T00Z => 2009-01-03T00Z => 2009-01-02T00Z => 2009-01-03T00Z =>
> 2009-01-02T00Z => 2009-01-03T00Z => 2009-01-02T00Z => 2009-01-05T00Z)
> 
> This is very strange... am I missing something ?
> 
> I also tried to redefine the time variable in combination with the ncoords
> attribute at different levels of the aggregations but the behavior were
> differents and not those expected, I can expose these if needed.
> 
> Now a last question, maybe the most important .. does the way I do to
> achieve this is the right one and if not how could I do ?


currently i dont think you can "cut off" the aggregation using the ncoords 
element. perhaps you should get things working without that first? also, i 
would use the latest 4.1 release. Its not yet stable, but has many bugs fixed. 
If you do see a bug, we will likely only fix it on 4.1.




> 
> Best regards
> 
> Jonathan
> 
> 


Ticket Details
===================
Ticket ID: JWA-773791
Department: Support THREDDS
Priority: Normal
Status: Open


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.