[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Problems with GRIB feature collections

Greetings Roy,

Not sure if this is causing the issues or not, but I noticed that the
collection from 201604:

http://thredds1.pfeg.noaa.gov/thredds/catalog/FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604/catalog.html?dataset=FNMOC/NAVGEM/onedegree/pressure/6hr/FNMOC_NAVGEM_1-degree_6-hourly_Pressure-201604

shows

Start: 2016-04-01T00:00:00Z
End: 2016-05-04T06:00:00Z

The end time in May looks a little suspicious. Can you check those
files to see if there is actually data from May in that directory? I
know the docs say that the directories should only contain grids from
the same partition, but I am not sure what happens if that rule is
violated (i.e. a few May grids in the April directory). It's a shot in
the dark.

Worst case - how big is the entire collection?

Cheers,

Sean


On Fri, Dec 8, 2017 at 10:07 AM, Roy Mendelssohn - NOAA Federal
<address@hidden> wrote:
> Hi Sean:
>
> So,  I shutdown the TDS.  I removed all the ncx* and gbx* files.  I removed 
> the grid cache.  So the restart should be completely clean.  I started the 
> TDM.  I started the TDS.  The TDM did its thing.  For the NAVGEM 1-day,  no 
> matter which of the best options I look at(Best,  or Full), the last time 
> still have an increment of  8 hours instead of 6,  and is off into the 
> future.  All of the other times are correct.
>
> Thanks,
>
> -Roy
>
>> On Dec 6, 2017, at 3:49 PM, Sean Arms <address@hidden> wrote:
>>
>> Hi Roy,
>>
>> I can't reproduce the issue here locally with the files and the
>> catalog you provided. Can you try to remove all of the .gbx9 and .ncx3
>> files for the one degree collection and have the TDM rebuild them from
>> scratch?
>>
>> Thanks!
>>
>> Sean
>>
>> On Wed, Dec 6, 2017 at 4:12 PM, Sean Arms <address@hidden> wrote:
>>> Hi Roy,
>>>
>>> It made it - thanks! Let me take a look and see what I come up with.
>>>
>>> Cheers!
>>>
>>> Sean
>>>
>>> On Wed, Dec 6, 2017 at 3:08 PM, Roy Mendelssohn - NOAA Federal
>>> <address@hidden> wrote:
>>>> Just sent an email with the file.  I know in the past sometimes files that 
>>>> make it through our mailer don't make it through yours.  Let me know if 
>>>> you get it,  otherwise I will send by a different method.
>>>>
>>>> -Roy
>>>>
>>>>> On Dec 6, 2017, at 1:36 PM, Sean Arms <address@hidden> wrote:
>>>>>
>>>>> Can I get a copy of the files from December 4th? If that is too large,
>>>>> I can work with the gbx9 and ncx3 files.
>>>>>
>>>>> Thanks!
>>>>>
>>>>> Sean
>>>>>
>>>>>
>>>>> On Wed, Dec 6, 2017 at 2:30 PM, Roy Mendelssohn - NOAA Federal
>>>>> <address@hidden> wrote:
>>>>>> The latest time range is correct based on the files in the directory.    
>>>>>> The files are in sub directories like 201712, so given that the xml 
>>>>>> defines "as directory"  it is doing that and is finding the correct 
>>>>>> times, if that helps at all.
>>>>>>
>>>>>> -Roy
>>>>>>
>>>>>>> On Dec 6, 2017, at 1:19 PM, Sean Arms <address@hidden> wrote:
>>>>>>>
>>>>>>> Hi Roy,
>>>>>>>
>>>>>>> This might take a bit. If we look at the time range on the "Best"
>>>>>>> collection, we get:
>>>>>>>
>>>>>>> Start: 2013-02-21T00:00:00Z
>>>>>>> End: 2017-12-09T06:00:00Z
>>>>>>>
>>>>>>> However, if we look at the time range on the "latest" (201712), we get:
>>>>>>>
>>>>>>> Start: 2017-12-01T00:00:00Z
>>>>>>> End: 2017-12-04T18:00:00Z
>>>>>>>
>>>>>>> It's not clear where anything after 2017-12-04T18:00:00Z is coming
>>>>>>> from, but I think I have a way of tracking it down if it isn't a bug
>>>>>>> (again, it will just take a bit of time).
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Sean
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Dec 6, 2017 at 12:47 PM, Roy Mendelssohn - NOAA Federal
>>>>>>> <address@hidden> wrote:
>>>>>>>> Okay the TwoD datasets are showing.  I still see the same issue with 
>>>>>>>> the last time.  Can easily send you sample data if that would help.
>>>>>>>>
>>>>>>>> -ROy
>>>>>>>>
>>>>>>>>> On Dec 6, 2017, at 11:38 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>
>>>>>>>>> Hmmm. Can you try removing the ncx files to force the TDM to 
>>>>>>>>> regenerate them? Both the TDS and TDM will need to be restarted.
>>>>>>>>>
>>>>>>>>> Cheers,
>>>>>>>>>
>>>>>>>>> Sean
>>>>>>>>>
>>>>>>>>> On Wed, Dec 6, 2017 at 12:17 PM Roy Mendelssohn - NOAA Federal 
>>>>>>>>> <address@hidden> wrote:
>>>>>>>>> Adding that appears to have sent that dataset into never-neverland.  
>>>>>>>>> Now I get no response.  DOn't see an error in the log file either.
>>>>>>>>>
>>>>>>>>> -Roy
>>>>>>>>>
>>>>>>>>>> On Dec 6, 2017, at 10:53 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>
>>>>>>>>>> Greetings Roy,
>>>>>>>>>>
>>>>>>>>>> Sorry for the delay - I've been in a teleconference all morning.
>>>>>>>>>>
>>>>>>>>>> So the TDS may be picking up on the fact that some of those records
>>>>>>>>>> are encoded as forecasts. That's possible, and I'll keep that in 
>>>>>>>>>> mind.
>>>>>>>>>>
>>>>>>>>>> The xml snippet should look like this (change made to first line to 
>>>>>>>>>> add TwoD):
>>>>>>>>>>
>>>>>>>>>> <gribConfig datasetTypes="TwoD Best LatestFile">
>>>>>>>>>> <pdsHash>
>>>>>>>>>>     <useTableVersion>false</useTableVersion>
>>>>>>>>>> </pdsHash>
>>>>>>>>>> </gribConfig>
>>>>>>>>>>
>>>>>>>>>> Cheers,
>>>>>>>>>>
>>>>>>>>>> Sean
>>>>>>>>>>
>>>>>>>>>> On Wed, Dec 6, 2017 at 11:05 AM, Roy Mendelssohn - NOAA Federal
>>>>>>>>>> <address@hidden> wrote:
>>>>>>>>>>> BTW - I should add that the 1-degree data is a special run that 
>>>>>>>>>>> they do for us,  and the way I understand it in their processing 
>>>>>>>>>>> terms the 00 and 12 files are considered real-time,  while the 06 
>>>>>>>>>>> and 18 files are considered "forecasts".  They have different 
>>>>>>>>>>> filenames because of that, for example:
>>>>>>>>>>>
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>>
>>>>>>>>>>> That is why in the regex for the time I have to skip two spaces.  
>>>>>>>>>>> In case this info helps any:
>>>>>>>>>>>
>>>>>>>>>>> wgrib -V 
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0RL2017120400_0102_000000-000000pres
>>>>>>>>>>> rec 1:21:date 2017120400 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 
>>>>>>>>>>> 0 missing 0
>>>>>>>>>>> center 58 subcenter 0 process 18 Table 2 scan: WE:SN winds(N/S)
>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>       long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>> min/max data 95236 104736  num bits 10  BDS_Ref 9523.6  DecScale -1 
>>>>>>>>>>> BinScale 0
>>>>>>>>>>>
>>>>>>>>>>> wgrib -V 
>>>>>>>>>>> US058GMET-GR1mdl.0018_0240_00000F0OF2017120418_0102_000000-000000pres
>>>>>>>>>>> rec 1:0:date 2017120418 PRMSL kpds5=2 kpds6=102 kpds7=0 
>>>>>>>>>>> levels=(0,0) grid=240 MSL anl:
>>>>>>>>>>> PRMSL=Pressure reduced to MSL [Pa]
>>>>>>>>>>> timerange 0 P1 0 P2 0 TimeU 1  nx 360 ny 181 GDS grid 0 num_in_ave 
>>>>>>>>>>> 0 missing 0
>>>>>>>>>>> center 58 subcenter 0 process 18 Table 3 scan: WE:SN winds(N/S)
>>>>>>>>>>> latlon: lat  -90.000000 to 90.000000 by 1.000000  nxny 65160
>>>>>>>>>>>       long 0.000000 to -1.000000 by 1.000000, (360 x 181) scan 64 
>>>>>>>>>>> mode 128 bdsgrid 1
>>>>>>>>>>> min/max data 94770 104260  num bits 10  BDS_Ref 9477  DecScale -1 
>>>>>>>>>>> BinScale 0
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> BTW - Note the different table,  which is why I have include in the 
>>>>>>>>>>> xml the stuff about:
>>>>>>>>>>>
>>>>>>>>>>>        <pdsHash>
>>>>>>>>>>>           <useTableVersion>false</useTableVersion>
>>>>>>>>>>>        </pdsHash>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>>
>>>>>>>>>>> -Roy
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> On Dec 6, 2017, at 6:09 AM, Sean Arms <address@hidden> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> For the second question regarding "full" vs "best", this has to do 
>>>>>>>>>>>> with how the overall collection is aggregated. For the half-degree 
>>>>>>>>>>>> NAVGEM, you have monthly collections of grids, but none of the 
>>>>>>>>>>>> forecasts overlap. Therefore, the overall collection aggregation 
>>>>>>>>>>>> exposes all of the grids. For the one degree, it looks like the 
>>>>>>>>>>>> TDS thinks there was an overlap in the grids (i.e. two grids with 
>>>>>>>>>>>> the same valid time), and so one of the overlapped grids was 
>>>>>>>>>>>> "better" (i.e. closer to a runtime) and the other was dropped to 
>>>>>>>>>>>> make a "best" collection. However, I don't see that happening in 
>>>>>>>>>>>> any of the monthly collections listed, so it is probably a bug. In 
>>>>>>>>>>>> the xml for these two collections, can you add "TwoD" to the 
>>>>>>>>>>>> datasetTypes attribute of the gribConfig element? That will give 
>>>>>>>>>>>> us a bit more information.
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> **********************
>>>>>>>>>>> "The contents of this message do not reflect any position of the 
>>>>>>>>>>> U.S. Government or NOAA."
>>>>>>>>>>> **********************
>>>>>>>>>>> Roy Mendelssohn
>>>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>>>> NOAA/NMFS
>>>>>>>>>>> Environmental Research Division
>>>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>>>> ***Note new street address***
>>>>>>>>>>> 110 McAllister Way
>>>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>>>
>>>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>>>> "the arc of the moral universe is long, but it bends toward 
>>>>>>>>>>> justice" -MLK Jr.
>>>>>>>>>>>
>>>>>>>>>
>>>>>>>>> **********************
>>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>>> Government or NOAA."
>>>>>>>>> **********************
>>>>>>>>> Roy Mendelssohn
>>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>>> NOAA/NMFS
>>>>>>>>> Environmental Research Division
>>>>>>>>> Southwest Fisheries Science Center
>>>>>>>>> ***Note new street address***
>>>>>>>>> 110 McAllister Way
>>>>>>>>> Santa Cruz, CA 95060
>>>>>>>>> Phone: (831)-420-3666
>>>>>>>>> Fax: (831) 420-3980
>>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>>
>>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>>> -MLK Jr.
>>>>>>>>>
>>>>>>>>
>>>>>>>> **********************
>>>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>>>> Government or NOAA."
>>>>>>>> **********************
>>>>>>>> Roy Mendelssohn
>>>>>>>> Supervisory Operations Research Analyst
>>>>>>>> NOAA/NMFS
>>>>>>>> Environmental Research Division
>>>>>>>> Southwest Fisheries Science Center
>>>>>>>> ***Note new street address***
>>>>>>>> 110 McAllister Way
>>>>>>>> Santa Cruz, CA 95060
>>>>>>>> Phone: (831)-420-3666
>>>>>>>> Fax: (831) 420-3980
>>>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>>>
>>>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>>>> "From those who have been given much, much will be expected"
>>>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>>>> -MLK Jr.
>>>>>>>>
>>>>>>
>>>>>> **********************
>>>>>> "The contents of this message do not reflect any position of the U.S. 
>>>>>> Government or NOAA."
>>>>>> **********************
>>>>>> Roy Mendelssohn
>>>>>> Supervisory Operations Research Analyst
>>>>>> NOAA/NMFS
>>>>>> Environmental Research Division
>>>>>> Southwest Fisheries Science Center
>>>>>> ***Note new street address***
>>>>>> 110 McAllister Way
>>>>>> Santa Cruz, CA 95060
>>>>>> Phone: (831)-420-3666
>>>>>> Fax: (831) 420-3980
>>>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>>>
>>>>>> "Old age and treachery will overcome youth and skill."
>>>>>> "From those who have been given much, much will be expected"
>>>>>> "the arc of the moral universe is long, but it bends toward justice" 
>>>>>> -MLK Jr.
>>>>>>
>>>>
>>>> **********************
>>>> "The contents of this message do not reflect any position of the U.S. 
>>>> Government or NOAA."
>>>> **********************
>>>> Roy Mendelssohn
>>>> Supervisory Operations Research Analyst
>>>> NOAA/NMFS
>>>> Environmental Research Division
>>>> Southwest Fisheries Science Center
>>>> ***Note new street address***
>>>> 110 McAllister Way
>>>> Santa Cruz, CA 95060
>>>> Phone: (831)-420-3666
>>>> Fax: (831) 420-3980
>>>> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>>>>
>>>> "Old age and treachery will overcome youth and skill."
>>>> "From those who have been given much, much will be expected"
>>>> "the arc of the moral universe is long, but it bends toward justice" -MLK 
>>>> Jr.
>>>>
>
> **********************
> "The contents of this message do not reflect any position of the U.S. 
> Government or NOAA."
> **********************
> Roy Mendelssohn
> Supervisory Operations Research Analyst
> NOAA/NMFS
> Environmental Research Division
> Southwest Fisheries Science Center
> ***Note new street address***
> 110 McAllister Way
> Santa Cruz, CA 95060
> Phone: (831)-420-3666
> Fax: (831) 420-3980
> e-mail: address@hidden www: http://www.pfeg.noaa.gov/
>
> "Old age and treachery will overcome youth and skill."
> "From those who have been given much, much will be expected"
> "the arc of the moral universe is long, but it bends toward justice" -MLK Jr.
>