Hey Sean,
Glad we were able to help find that bug, but I don't think the "project"
variable (or lack of) is the root of our problem as i chose your option 3
(my mistake, this was supposed to be the same after the last one) and i
have similar response. Good news, when i add the #noprefetch option, it
seems to fix it. hopefully this helps provide answers. Demonstrated by
below code.
# Failure with python (matlab as well)
import netCDF4 as nc
for url in urls:
print(nc.Dataset(url)['time'])
variables= nc.Dataset(url).variables.keys()
for var in variables:
try:
nc.Dataset(url)[var][0]
print('Success! {} from {}'.format(var, url))
except IndexError:
print("won't load variable {} from {}".format(var, url))
url += "#noprefetch"
nc.Dataset(url)[var][0]
print('Success! {} from {}'.format(var, url))
except IndexError as e:
print("FAIL: load variable {} from {}".format(var, url))
print(' {}'.format(e))
On Fri, Jan 31, 2020 at 4:25 PM Sean Arms <sarms@xxxxxxxx> wrote:
> I thought it was working for me, but for the wrong reasons. Sorry about
> that. But, now I have it.
>
> The error message from the server is...well...garbage. That's something we
> need to look into. The underlying problem here is that the new data file
> does not include the variable project. Because variable "project" exists in
> the other files (or more specifically, the first file of the aggregation),
> and specifically because it has time as it's outer dimension, currently it
> must exist in all files of the aggregation. So, we can make an ascii
> request for that variable specifically, and request everything up to the
> last value, and it works [0:1:424]:
>
>
> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B0:1:424%5D
>
>
> However, once we ask for the last value, it bombs [0:1:424] :
>
>
> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B0:1:425%5D
>
> It's only like this for the full read code path, though. If I slice it
> from [1:1:424] (skip the first value), it works
>
>
> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?project%5B1:1:425%5D
>
> ...and the missing value gets filled with a zero, because why not (well,
> ok, I can think of a few reasons). So, from the TDS side, and more
> specifically netCDF-Java, this is a bug (well, bugs, because returning zero
> isn't quite the thing to do here either).
>
> From what I can understand, netCDF-C tries to preload some data from
> remote servers (if the variable is considered "small enough"), and because
> the variable "project" is "small enough", the library tries to grab it all
> and the request bombs out. You can tell the C library to not do that by
> adding "#noprefetch" at the end of the url when opening the dataset, but
> that just delays things (unless you or the underlying library you are using
> never tries to fully read the variable "project"). So, in your sample
> python code, pass Netcdf4Python's Dataset the following URL and give it a
> spin:
>
>
> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml#noprefetch
>
> Cool...so what to do until I can get this fixed. Three things off the top
> of my head:
>
> 1. Add "#noprefetch" to your URL (and anyone else using the dataset). Barf.
> 2. Use NcML in the aggregation to remove the variable "project" all
> together, as it seems to be -999.0 when it does exist anyways.
> 3. Add the variable "project" to the latest file (NcML or by rewriting it).
>
> Sorry this took so long to debug. There will be a fix, it will just take
> some time (maybe early next week since I believe I know exactly what needs
> done and where).
>
> Cheers,
>
> Sean
>
>
> On Fri, Jan 31, 2020 at 12:27 PM Spicer Bak <spicer.bak.frf@xxxxxxxxx>
> wrote:
>
>> Hey Sean,
>> Thanks for getting back to me. I was still getting the same symptoms
>> earlier today. Did it work on second inquiry for you?
>>
>> I did make a change to that last file so the dim's match up with
>> previous. After I did that, checked again to make sure that wasn't causing
>> the problem. Seems i still am getting the same IndexError i was getting,
>> so this (as you mentioned) is not the root of the problem.
>>
>> On Fri, Jan 31, 2020 at 11:03 AM Sean Arms <sarms@xxxxxxxx> wrote:
>>
>>> Greetings Spicer,
>>>
>>> It looks like you have solved the issue? I was having problems the other
>>> day as well. I downloaded the files locally to see if I could reproduce the
>>> issue, and I am unable to do so now. However, I do see something that might
>>> cause an issue down the road. The dimension order on some of the variables
>>> in the latest file does not match what is in other files. For example, if
>>> we compare the latest file:
>>>
>>>
>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20200110_1180_FRF_NAVD88_LARC_GPS_UTC_v20200113_grid_latlon.nc.dds
>>>
>>> with next latest file:
>>>
>>>
>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/FRF_20191206_1179_FRF_NAVD88_LARC_GPS_UTC_v20191209_grid_latlon.nc.dds
>>>
>>> The dimensionality of latitude, longitude, northing, and easting switch
>>> from (xFRF, yFRF) (20191206) to (yFRF, xFRF) (20200113).Might not be a big
>>> issue overall since the use of an NcML file on the server (as opposed to
>>> NcML directly in the catalog) uses the first file of the aggregation as
>>> template.
>>>
>>> Cheers,
>>>
>>> Sean
>>>
>>>
>>> On Wed, Jan 29, 2020 at 3:03 PM Spicer Bak <spicer.bak.frf@xxxxxxxxx>
>>> wrote:
>>>
>>>> hello TDS community,
>>>> I have a problem I'm quite pickled on. We had a dataset that was
>>>> working fine until (i think) we pushed the latest file. Our server is
>>>> continually updated with new files and all of the other datasets seem to
>>>> work fine.
>>>>
>>>> I'm able to get the data from the individual file URLS, but not the
>>>> time concatenated ncml file. I can display the file or variables, but not
>>>> obtain any of the data through OPeNDAP, but i'm able to see the data
>>>> returned via the "get ASCII" button on the OPeNDAP page. The below python
>>>> script demonstrates the problem:
>>>>
>>>> # success with ncml:
>>>>
>>>> https://chldata.erdc.dren.mil/thredds/dodsC/frf/geomorphology/DEMs/surveyDEM/surveyDEM.ncml.ascii?latitude%5B0:1:0%5D%5B0:1:0%5D,time%5B0:1:425%5D,elevation%5B0:1:0%5D%5B0:1:0%5D%5B0:1:0%5D
>>>>
>>>> # Failure with python (matlab as well)
>>>> import netCDF4 as nc
>>>> for url in urls:
>>>> print(nc.Dataset(url)['time'])
>>>> variables= nc.Dataset(url).variables.keys()
>>>> for var in variables:
>>>> try:
>>>> nc.Dataset(url)[var][0]
>>>> print('Success! {} from {}'.format(var, url))
>>>> except IndexError as e:
>>>> print("won't load variable {} from {}".format(var, url))
>>>>
>>>> print(' {}'.format(e))
>>>>
>>>> Any help would be much appreciated!
>>>>
>>>> --
>>>> +++++++++++++++++++++++++++
>>>> Spicer Bak, PhD
>>>> USACE CHL Field Research Facility
>>>> 252-305-9975
>>>> _______________________________________________
>>>> NOTE: All exchanges posted to Unidata maintained email lists are
>>>> recorded in the Unidata inquiry tracking system and made publicly
>>>> available through the web. Users who post to any of the lists we
>>>> maintain are reminded to remove any personal information that they
>>>> do not want to be made public.
>>>>
>>>>
>>>> thredds mailing list
>>>> thredds@xxxxxxxxxxxxxxxx
>>>> For list information or to unsubscribe, visit:
>>>> https://www.unidata.ucar.edu/mailing_lists/
>>>>
>>>
>>
>> --
>> +++++++++++++++++++++++++++
>> Spicer Bak, PhD
>> USACE CHL Field Research Facility
>> 252-305-9975
>>
>
--
+++++++++++++++++++++++++++
Spicer Bak, PhD
USACE CHL Field Research Facility
252-305-9975
