Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Due to the current gap in continued funding from the U.S. National Science Foundation (NSF), the NSF Unidata Program Center has temporarily paused most operations. See NSF Unidata Pause in Most Operations for details.
Hi Sean,So essentially, you're saying that it is the responsibility of the writer of the .ncml writer to put the files in the right order, they can't just specify the files in any order and expect netcdf-java to order them? Okay...
In that case, the issue is only with geotools which is assuming an increasing order (not decreasing).
Regards Niels On 06-10-16 21:18, Sean Arms wrote:
Hi Niles,So, quick question - do you expect the valid time in the file hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc> to be before or after the valid time in hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>?Here is what I could tell from the actual data in the files:hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>: time -> 2014-07-08T00:00:00Z (127248.0 hours since 2000-01-01) hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>: time -> 2014-07-07T03:00:00Z (127227.0 hours since 2000-01-01)hycom_glb_regp01_t000.nc <http://hycom_glb_regp01_t000.nc>: time -> 2014-07-08T00:00:00Z(127248.0 hours since 2000-01-01) hycom_glb_regp01_t003.nc <http://hycom_glb_regp01_t003.nc>: time -> 2014-07-07T03:00:00Z (127227.0 hours since 2000-01-01)Therefor, an aggregation listing the order such as: hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc> hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>will result in times decreasing in values (2014-07-08T00:00:00Z, 2014-07-07T03:00:00Z) because that's what is in the actual data files. However, if the expectation is that the valid time in _t003 is after the valid time in _t000, then things might only look correct if you reverse the aggregation order.SeanOn Thu, Oct 6, 2016 at 12:33 PM, Sean Arms <sarms@xxxxxxxx <mailto:sarms@xxxxxxxx>> wrote:Hi Niles, If you specify individual files for aggregation, they will be in the order you specify (which is exactly what you have found). If you instead to a scan to aggregate the file, the files will be aggregated in lexicographical order (for example, the order shown by the command `ls`). This may or may not be in the appropriate order. This is one of many tricky things about using NcML aggregations. This is also one of the motivating reasons behind the concept of featureCollections in the TDS. In a featureCollection (Forecast Model Run Collections (FMRC), for example), the coordinate systems of all of the files in a collection are examined, and the TDS tries to "do the right thing" with them with minimal configuration. Cheers, Sean On Wed, Oct 5, 2016 at 3:32 AM, Niels Charlier <niels@xxxxxxxxx <mailto:niels@xxxxxxxxx>> wrote: Following this question, what is going to happen in this regard if you do a <scan> to list the aggregation files? Will they be put in an appropriate order? Thanks for your answer. Regards Niels On 09/30/2016 12:16 PM, Niels Charlier wrote: Hi Everyone, I have changed my mind on this other aggregation issue I discussed in one of my emails (see below), I said it was unrelated to netcdf-java but it doesn't seem to be the case. Well, I really want to hear your opinion. The issue is related to the order of files listed for aggregations. I have found that the issue lies in part with geotools, because it is making certain assumptions about time variables being increasing rather than decreasing. Convention does demand that a variable is either one of the two (but not in random order). The question is whether the list of files in an .ncml files should also be in a certain order. It seems that netcdf-java (which does the actual aggregating of the variables) follows the order from the ncml code. That means that the values inside the aggregated variable may end up in random order. For example, assume you have two files. One has time values 1,2,3. The the other one has 4,5,6. If we now place them in opposite order in our .ncml file, netcdf-java appears to create an aggregated variable with values 4,5,6,1,2,3 In that case, geotools cannot even assume that the extremes are on both ends of the list, which is IMO faulty and can cause many issues with other assumptions in the code. Have I found something here, or do you disagree? Thank you for responding. Should we just demand from users to place the list of file sin the same order as the order of values inside the .nc files (increasing/decreasing)? is that (un)reasonable? Kind Regards Niels On 08-09-16 14:51, Niels Charlier wrote: 3. changing the order of the list of files in the inner aggregation tags. <netcdf location="hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>"/> <netcdf location="hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>"/> instead of <netcdf location="hycom_glb_regp01_old_t000.nc <http://hycom_glb_regp01_old_t000.nc>"/> <netcdf location="hycom_glb_regp01_old_t003.nc <http://hycom_glb_regp01_old_t003.nc>"/> Otherwise the one with the earliest time coordinates is last in the list instead of first... and geotools fails on this. Is this normal? This issue is definitely unrelated to netcdf-java, but I am not sure whether this is a bug in geotools or in the .ncml file. _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdf-java mailing list netcdf-java@xxxxxxxxxxxxxxxx <mailto:netcdf-java@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ <http://www.unidata.ucar.edu/mailing_lists/> _______________________________________________ NOTE: All exchanges posted to Unidata maintained email lists are recorded in the Unidata inquiry tracking system and made publicly available through the web. Users who post to any of the lists we maintain are reminded to remove any personal information that they do not want to be made public. netcdf-java mailing list netcdf-java@xxxxxxxxxxxxxxxx <mailto:netcdf-java@xxxxxxxxxxxxxxxx> For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/ <http://www.unidata.ucar.edu/mailing_lists/>
netcdf-java archives: