[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problems with GRIB feature collections



Greetings Roy,

Not sure if this is causing the issues or not, but I noticed that the
collection from 201604:

http://thredds1.pfeg.noaa.gov/thredds/catalog/FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604/catalog.html?dataset=FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604

shows

Start: 2016-04-01T00:00:00Z
End: 2016-05-04T06:00:00Z

The end time in May looks a little suspicious. Can you check those
files to see if there is actually data from May in that directory? I
know the docs say that the directories should only contain grids from
the same partition, but I am not sure what happens if that rule is
violated (i.e. a few May grids in the April directory). It's a shot in
the dark.

Worst case - how big is the entire collection?

Cheers,

Sean


On Fri, Dec 8, 2017 at 10:07 AM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> Hi Sean:
>
> So,  I shutdown the TDS.  I removed all the ncx* and gbx* files.  I removed 
> the grid cache.  So the restart should be completely clean.  I started the 
> TDM.  I started the TDS.  The TDM did its thing.  For the NAVGEM 1-day,  no 
> matter which of the best options I look at(Best,  or Full), the last time 
> still have an increment of  8 hours instead of 6,  and is off into the 
> future.  All of the other times are correct.
>
> Thanks,
>
> -Roy
>
>> On Dec 6, 2017, at 3:49 PM, Sean Arms <address@hidden> wrote:
>>
>> Hi Roy,
>>
>> I can't reproduce the issue here locally with the files and the
>> catalog you provided. Can you try to remove all of the .gbx9 and .ncx3
>> files for the one degree collection and have the TDM rebuild them from
>> scratch?
>>
>> Thanks!
>>
>> Sean
>>
>> On Wed, Dec 6, 2017 at 4:12 PM, Sean Arms <address@hidden> wrote:
>>> Hi Roy,
>>>
>>> It made it - thanks! Let me take a look and see what I come up with.
>>>
>>> Cheers!
>>>
>>> Sean
>>>
>>> On Wed, Dec 6, 2017 at 3:08 PM, Roy Mendelssohn - NOAA Federal
>>> <address@hidden> wrote:
>>>> Just sent an email with the file.  I know in the past sometimes files that 
>>>> make it through our mailer don't make it through yours.  Let me know if 
>>>> you get it,  otherwise I will send by a different method.
>>>>
>>>> -Roy
>>>>
>>>>> On Dec 6, 2017, at 1:36 PM, Sean Arms <address@hidden> wrote:
>>>>>
>>>>> Can I get a copy of the files from December 4th? If that is too large,
>>>>> I can work with the gbx9 and ncx3 files.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Sean
>>>>>
>>>>>
>>>>> On Wed, Dec 6, 2017 at 2:30 PM, Roy Mendelssohn - NOAA Federal
>>>>> <address@hidden> wrote:
>>>>>> The latest time range is correct based on the files in the directory.    
>>>>>> The files are in sub directories like 201712, so given that the xml 
>>>>>> defines "as directory"  it is doing that and is finding the correct 
>>>>>> times, if that helps at all.
>>>>>>
>>>>>> -Roy
>>>>>>
>>>>>>> On Dec 6, 2017, at 1:19 PM, Sean Arms <address@hidden> wrote:
>>>>>>>
>>>>>>> Hi Roy,
>>>>>>>
>>>>>>> This might take a bit. If we look at the time range on the "Best"
>>>>>>> collection, we get:
>>>>>>>
>>>>>>> Start: 2013-02-21T00:00:00Z
>>>>>>> End: 2017-12-09T06:00:00Z
>>>>>>>
>>>>>>> However, if we look at the time range on the "latest" (201712), we get:
>>>>>>>
>>>>>>> Start: 2017-12-01T00:00:00Z
>>>>>>> End: 2017-12-04T18:00:00Z
>>>>>>>
>>>>>>> It's not clear where anything after 2017-12-04T18:00:00Z is coming
>>>>>>> from, but I think I have a way of tracking it down if it isn't a bug
>>>>>>> (again, it will just take a bit of time).
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Sean
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
>>>>>>> <address@hidden> wrote:
>>>>>>>> Okay the TwoD datasets are showing.  I still see the same issue with 
>>>>>>>> the last time.  Can easily send you sample data if that would help.
>>>>>>>>
>>>>>>>> -ROy
>>>>>>>>
>>>>>>>>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>
>>>>>>>>> Hmmm. Can you try removing the ncx files to force the TDM to 
>>>>>>>>> regenerate them? Both the TDS and TDM will need to be restarted.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Sean
>>>>>>>>>
>>>>>>>>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>>>>>>>>> <address@hidden> wrote:
>>>>>>>>> Adding that appears to have sent that dataset into never-neverland.  
>>>>>>>>> Now I get no response.  DOn't see an error in the log file either.
>>>>>>>>>
>>>>>>>>> -Roy
>>>>>>>>>
>>>>>>>>>> On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>
>>>>>>>>>> Greetings Roy,
>>>>>>>>>>
>>>>>>>>>> Sorry for the delay - I've been in a teleconference all morning.
>>>>>>>>>>
>>>>>>>>>> So the TDS may be picking up on the fact that some of those records
>>>>>>>>>> are encoded as forecasts. That's possible, and I'll keep that in 
>>>>>>>>>> mind.
>>>>>>>>>>
>>>>>>>>>> The xml snippet should look like this (change made to first line to 
>>>>>>>>>> add TwoD):
>>>>>>>>>>
>>>>>>>>>> <gribConfig datasetTypes="TwoD Best LatestFile">
>>>>>>>>>> <pdsHash>
>>>>>>>>>>     <useTableVersion>false</useTableVersion>
>>>>>>>>>> </pdsHash>
>>>>>>>>>> </gribConfig>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Sean
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>>>>>>>>>> <address@hidden> wrote:
>>>>>>>>>>> BTW - I should add that the 1-degree data is a special run that 
>>>>>>>>>>> they do for us,  and the way I understand it in their processing 
>>>>>>>>>>> terms the 00 and 12 files are considered real-time,  while the 06 
>>>>>>>>>>> and 18 files are considered "forecasts".  They have different 
>>>>>>>>>>> filenames because of that, for example:
>>>>>>>>>>>
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>>
>>>>>>>>>>> That is why in the regex for the time I have to skip two spaces.  
>>>>>>>>>>> In case this info helps any:
>>>>>>>>>>>
>>>>>>>>>>> wgrib -V 
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 
>>>>>>>>>>> 0 missing 0
>>>>>>>>>>> center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>       long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>> min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale -1 
>>>>>>>>>>> BinScale 0
>>>>>>>>>>>
>>>>>>>>>>> wgrib -V 
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 
>>>>>>>>>>> 0 missing 0
>>>>>>>>>>> center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>       long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>> min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>>>>>>>>>>> BinScale 0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> BTW - Note the different table,  which is why I have include in the 
>>>>>>>>>>> xml the stuff about:
>>>>>>>>>>>
>>>>>>>>>>>        <pdsHash>
>>>>>>>>>>>           <useTableVersion>false</useTableVersion>
>>>>>>>>>>>        </pdsHash>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> -Roy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> For the second question regarding "full" vs "best", this has to do 
>>>>>>>>>>>> with how the overall collection is aggregated. For the half-degree 
>>>>>>>>>>>> NAVGEM, you have monthly collections of grids, but none of the 
>>>>>>>>>>>> forecasts overlap. Therefore, the overall collection aggregation 
>>>>>>>>>>>> exposes all of the grids. For the one degree, it looks like the 
>>>>>>>>>>>> TDS thinks there was an overlap in the grids (i.e. two grids with 
>>>>>>>>>>>> the same valid time), and so one of the overlapped grids was 
>>>>>>>>>>>> "better" (i.e. closer to a runtime) and the other was dropped to 
>>>>>>>>>>>> make a "best" collection. However, I don't see that happening in 
>>>>>>>>>>>> any of the monthly collections listed, so it is probably a bug. In 
>>>>>>>>>>>> the xml for these two collections, can you add "TwoD" to the 
>>>>>>>>>>>> datasetTypes attribute of the gribConfig element? That will give 
>>>>>>>>>>>> us a bit more information.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> **********************
>>>>>>>>>>> "The contents of this message do not reflect any position of the 
>>>>>>>>>>> U.S. Government or NOAA."
>>>>>>>>>>> **********************
>>>>>>>>>>> Roy Mendelssohn
>>>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>>>> NOAA/NMFS
>>>>>>>>>>> Environmental Research Division
>>>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>>>> ***Note new street address***
>>>>>>>>>>> 110 McAllister Way
>>>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>>>
>>>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>>>> "the arc of the moral universe is long, but it bends toward 
>>>>>>>>>>> justice" -MLK Jr.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> **********************
>>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>>> Government or NOAA."
>>>>>>>>> **********************
>>>>>>>>> Roy Mendelssohn
>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>> NOAA/NMFS
>>>>>>>>> Environmental Research Division
>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>> ***Note new street address***
>>>>>>>>> 110 McAllister Way
>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>
>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>>> -MLK Jr.
>>>>>>>>>
>>>>>>>>
>>>>>>>> **********************
>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>> Government or NOAA."
>>>>>>>> **********************
>>>>>>>> Roy Mendelssohn
>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>> NOAA/NMFS
>>>>>>>> Environmental Research Division
>>>>>>>> Southwest Fisheries Science Center
>>>>>>>> ***Note new street address***
>>>>>>>> 110 McAllister Way
>>>>>>>> Santa Cruz, CA 95060
>>>>>>>> Phone: (831)-420-3666
>>>>>>>> Fax: (831) 420-3980
>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>
>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>> -MLK Jr.
>>>>>>>>
>>>>>>
>>>>>> **********************
>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>> Government or NOAA."
>>>>>> **********************
>>>>>> Roy Mendelssohn
>>>>>> Supervisory Operations Research Analyst
>>>>>> NOAA/NMFS
>>>>>> Environmental Research Division
>>>>>> Southwest Fisheries Science Center
>>>>>> ***Note new street address***
>>>>>> 110 McAllister Way
>>>>>> Santa Cruz, CA 95060
>>>>>> Phone: (831)-420-3666
>>>>>> Fax: (831) 420-3980
>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>
>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>> "From those who have been given much, much will be expected"
>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>> -MLK Jr.
>>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. 
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> ***Note new street address***
>>>> 110 McAllister Way
>>>> Santa Cruz, CA 95060
>>>> Phone: (831)-420-3666
>>>> Fax: (831) 420-3980
>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>>> Jr.
>>>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>


NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.