Re: [netcdf-java] order of aggregation

  • To: Niels Charlier <niels@xxxxxxxxx>
  • Subject: Re: [netcdf-java] order of aggregation
  • From: Sean Arms <sarms@xxxxxxxx>
  • Date: Thu, 6 Oct 2016 12:33:47 -0600
Hi Niles,

If you specify individual files for aggregation, they will be in the order
you specify (which is exactly what you have found). If you instead to a
scan to aggregate the file, the files will be aggregated in lexicographical
order (for example, the order shown by the command `ls`). This may or may
not be in the appropriate order. This is one of many tricky things about
using NcML aggregations. This is also one of the motivating reasons behind
the concept of featureCollections in the TDS. In a featureCollection
(Forecast Model Run Collections (FMRC), for example), the coordinate
systems of all of the files in a collection are examined, and the TDS tries
to "do the right thing" with them with minimal configuration.

Cheers,

Sean

On Wed, Oct 5, 2016 at 3:32 AM, Niels Charlier <niels@xxxxxxxxx> wrote:

> Following this question, what is going to happen in this regard if you do
> a <scan> to list the aggregation files? Will they be put in an appropriate
> order?
>
> Thanks for your answer.
>
> Regards
>
> Niels
>
>
> On 09/30/2016 12:16 PM, Niels Charlier wrote:
>
>> Hi Everyone,
>>
>> I have changed my mind on this other aggregation issue I discussed in one
>> of my emails (see below), I said it was unrelated to netcdf-java but it
>> doesn't seem to be the case. Well, I really want to hear your opinion. The
>> issue is related to the order of files listed for aggregations.
>>
>> I have found that the issue lies in part with geotools, because it is
>> making certain assumptions about time variables being increasing rather
>> than decreasing. Convention does demand that a variable is either one of
>> the two (but not in random order).
>>
>> The question is whether the list of files in an .ncml files should also
>> be in a certain order. It seems that netcdf-java (which does the actual
>> aggregating of the variables) follows the order from the ncml code. That
>> means that the values inside the aggregated variable may end up in random
>> order.
>>
>> For example, assume you have two files.
>> One has time values 1,2,3. The the other one has 4,5,6.
>>
>> If we now place them in opposite order in our .ncml file, netcdf-java
>> appears to create an aggregated variable with values 4,5,6,1,2,3
>>
>> In that case, geotools cannot even assume that the extremes are on both
>> ends of the list, which is IMO faulty and can cause many issues with other
>> assumptions in the code.
>>
>> Have I found something here, or do you disagree? Thank you for responding.
>>
>> Should we just demand from users to place the list of file sin the same
>> order as the order of values inside the .nc files (increasing/decreasing)?
>> is that (un)reasonable?
>>
>> Kind Regards
>> Niels
>>
>> On 08-09-16 14:51, Niels Charlier wrote:
>>
>>> 3. changing the order of the list of files in the inner aggregation tags.
>>>                         <netcdf location="hycom_glb_regp01_old_t003.nc"/>
>>> <netcdf location="hycom_glb_regp01_old_t000.nc"/>
>>>             instead of  <netcdf location="hycom_glb_regp01_old_t000.nc"/>
>>> <netcdf location="hycom_glb_regp01_old_t003.nc"/>
>>>    Otherwise the one with the earliest time coordinates is last in the
>>> list instead of first... and geotools fails on this. Is this normal?
>>>
>>>    This issue is definitely unrelated to netcdf-java, but I am not sure
>>> whether this is a bug in geotools or in the .ncml file.
>>>
>>
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>>
>>
>> netcdf-java mailing list
>> netcdf-java@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe, visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> _______________________________________________
> NOTE: All exchanges posted to Unidata maintained email lists are
> recorded in the Unidata inquiry tracking system and made publicly
> available through the web.  Users who post to any of the lists we
> maintain are reminded to remove any personal information that they
> do not want to be made public.
>
>
> netcdf-java mailing list
> netcdf-java@xxxxxxxxxxxxxxxx
> For list information or to unsubscribe, visit:
> http://www.unidata.ucar.edu/mailing_lists/
>
  • 2016 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the netcdf-java archives: