[ldm-users] INC0316836 has new comments ( GOES Imagery)(Active)

  • To: nco.ops@xxxxxxxx
  • Subject: [ldm-users] INC0316836 has new comments ( GOES Imagery)(Active)
  • From: NOAA ServiceNow Platform <nsdesk@xxxxxxxxxxxxxxxxxxxxxx>
  • Date: Thu, 16 Dec 2021 07:23:09 -0800 (PST)
  • Auto-submitted: auto-generated
<html><head></head><body><img src="cid:ncoEmailLogo.png@SNC.a8e002813250a116" 
height="50">&nbsp;&nbsp;&nbsp;&nbsp;&nbsp;<b>INC0316836 has new comments ( GOES 
Imagery)(Active)</b><hr>

<style>.hideInEmail { display: none; }; </style>
<div></div>
<div><p>Some details of the ticket are included below:</p><table><tbody 
style="vertical-align:top;"><tr><th 
style="min-width:150px;"></th><th></th></tr><tr><td><b>End 
User:</b></td><td>Jeff Kelley</td></tr><tr><td><b>Short 
description:</b></td><td> GOES Imagery</td></tr><tr><td><b>Customer Impact 
Statement:</b></td><td>missing tiles in GOES 
imagery</td></tr><tr><td><b>State:</b></td><td>Active</td></tr><tr><td><b>Priority:</b></td><td>High</td></tr><tr><td><b>Queue
 
position:</b></td><td>152</td></tr><tr><td><b>Category:</b></td><td>General</td></tr><tr><td><b>Sub-Category:</b></td><td>Other</td></tr><tr><td><b>Assignment
 group:</b></td><td>NCO OMB Tech 
Control</td></tr><tr><td><b>Description:</b></td><td>Received email 
notification from Jeff Kelley:  Hi NCO. Have you been in contact with NESDIS at 
all about the missing tiles in GOES-16? Every image that comes in has missing 
tiles, whether in AWIPS, the FAA weather system, or even the College of Dupage 
website. 
<br><br>https://weather.cod.edu/satrad/?parms=continental-conus-13-24-1-100-1&amp;checked=map&amp;colorbar=undefined<br><br>I
 haven't seen any ADA or ADM messages either.<br><br>Jeff<br>ZID 
CWSU</td></tr></tbody></table></div>
<div><p>The following comments were given about the Incident:</p><div 
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div 
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid 
Purple;margin:2px 0;"><strong>2021-12-16 10:22:47 EST - Guest </strong><sup 
style="float:right;">Additional comments</sup><div><span 
style="word-wrap:break-word;display:block;">reply from: 
gilbert@xxxxxxxxxxxxxxxx<br><br>This issue has started again as of 14:30Z this 
morning.</span></div></div></td></tr><tr><td><div style="padding:10px 15px; 
border:1px solid lightgray; border-left:7px solid Purple;margin:2px 
0;"><strong>2021-12-13 20:14:24 EST - Guest </strong><sup 
style="float:right;">Additional comments</sup><div><span 
style="word-wrap:break-word;display:block;">reply from: 
gilbert@xxxxxxxxxxxxxxxx<br><br>Someone from the College of DuPage reached out 
to me with an interesting observation from the University of 
Oklahoma...<br><br>We had an outage on our NOAAport receive system this 
morning, which caused our systems to fail over to our partner organization's 
feeds. On investigation, we found that our receiver, ingest software, and LDM 
software were all working fine. We saw packets coming into the Linux-based 
ingest server, but nothing was being ingested. Then, it was noticed that the 
source IP of the NOAAport multicast traffic had changed, for at least some of 
the packets. Whereas in the past, it had been 10.0.9.51 , we are now seeing 
packets from 10.0.9.52 . Our ingest server was not accepting those because it 
had no special routing rule for that IP. We added such a rule (and did *not* 
delete the existing one for the .51 IP) and our NOAAport ingest began working 
as before.<br><br>I can confirm the source IP for me is 10.0.9.52 as well, but 
unlike him that IP range was already open on my system and that traffic was 
making its way in.  I'm not sure if this is related or not, but I wanted to 
pass along the intel in case it is.</span></div></div></td></tr><tr><td><div 
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid 
Purple;margin:2px 0;"><strong>2021-12-13 19:54:36 EST - Jeff 
Kelley</strong><sup style="float:right;">Additional comments</sup><div><span 
style="word-wrap:break-word;display:block;">reply from: 
jeff.kelley@xxxxxxxx<br><br>I just got off the phone with NCF. They are aware 
that data dropouts are<br>occurring (with all data types, not just satellite) 
after switching between<br>systems earlier today. The tech mentioned that it 
might be a 
firewall<br>issue.<br><br>Jeff</span></div></div></td></tr><tr><td><div 
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid 
Purple;margin:2px 0;"><strong>2021-12-13 19:53:01 EST - Shellene 
Cope</strong><sup style="float:right;">Additional comments</sup><div><span 
style="word-wrap:break-word;display:block;">Gilbert Sebenste emailed:  1) I 
have been using the GOES Restitch for a couple of months now and it has worked 
perfectly until today. Today I am getting partial images. Upon looking at the 
log files I am finding that it is "Timing Out" waiting for the  tile to arrive. 
I am not seeing any large amounts of dropped packets. I am not sure what could 
be causing my sudden issue.  Any insight as to what I might check?
<br>
<br> 
<br>
<br>Thanks, Jim  ( partial log below)
<br>
<br> 
<br>
<br> 
<br>
<br>[TIRE 132051] 2021-12-13 20:54:52,378 [process_items]: 
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:53,177 [process_items]: 
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:54:54,033 [process_items]: 
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,091 [process_items]: 
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
11 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:00,753 [process_items]: 
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:01,554 [process_items]: 
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
12 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,066 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 6 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:06,350 [process_items]: 
OR_ABI-L2-CMIPC-M6C03_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
13 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,089 [process_items]: 
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:07,592 [process_items]: 
OR_ABI-L2-CMIPC-M6C01_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:08,514 [process_items]: 
OR_ABI-L2-CMIPC-M6C05_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
10 of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:13,149 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 7 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:16,471 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 8 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:22,188 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 9 
of 15 tiles received.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:34,664 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
10 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,834 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,944 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel07/20211213/OR_ABI-L2-CMIPC-M6C07_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:44,947 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:45,051 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel12/20211213/OR_ABI-L2-CMIPC-M6C12_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:45,698 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
11 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,772 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:47,875 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel10/20211213/OR_ABI-L2-CMIPC-M6C10_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,075 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:48,181 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel11/20211213/OR_ABI-L2-CMIPC-M6C11_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,455 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,559 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel09/20211213/OR_ABI-L2-CMIPC-M6C09_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,560 [process_items]: Finishing due to 
timeout.
<br>
<br>
<br>
<br>[TIRE 132046] 2021-12-13 20:55:49,664 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel08/20211213/OR_ABI-L2-CMIPC-M6C08_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>[TIRE 132051] 2021-12-13 20:55:49,891 [process_items]: 
OR_ABI-L2-CMIPC-M6C02_G16_s20213472051170_e20213472051170_c20213472051170.nc: 
12 of 15 tiles received.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,447 [process_items]: Finishing due to 
timeout.
<br>
<br>[TIRE 132046] 2021-12-13 20:55:51,556 [store_done]: 
GOES16/Products/CloudAndMoistureImagery/CONUS/Channel13/20211213/OR_ABI-L2-CMIPC-M6C13_G16_s20213472046170_e20213472046170_c20213472046170.nc
 finished.
<br>
<br>2) Hi Jim,
<br>
<br>I was just about to write my own email about this.  It seems to me that not 
all GOES-16 tiles are making it to NOAAPort.  I was about to ask if anyone else 
is seeing this, and I guess you are.
<br>
<br>I noticed this began around 14Z today.  GOES-16 CONUS &amp; Full-Disk are 
impacted, Mesoscale and GOES-17 seem okay.  I haven't seen any announcements 
about data flow issues.  Using notifyme I saw that Unidata was missing the same 
tiles I was.
<br>
<br>Does anyone else have any insight?
<br>
<br>-Mike
<br>
<br>3) Hello NCO,
<br>
<br>Some GOES-16 tiles on NOAAport/SBN are missing. This started just after 14Z,
<br>when you awitched to the BMGS. See the thread below from a UNIDATA user log.
<br>Multiple people are seeing this.
<br>
<br>Gilbert Sebenste
<br>Meteorological Consultant
<br>AllisonHouse, LLC
<br>
<br>(SEC)</span></div></div></td></tr><tr><td><div style="padding:10px 15px; 
border:1px solid lightgray; border-left:7px solid Purple;margin:2px 
0;"><strong>2021-12-13 19:48:34 EST - Shellene Cope</strong><sup 
style="float:right;">Additional comments</sup><div><span 
style="word-wrap:break-word;display:block;">Jeff Kelley has emailed the 
following reply:  There has been no improvement in this issue. Have you heard 
anything from NESDIS? It's strange that the imagery on the STAR and NASA 
websites appear fine.
<br>
<br>https://weather.msfc.nasa.gov/cgi-bin/basicLooper.pl?category=goeseastabiconus&amp;regex=10p35um&amp;title=GOES-East%20ABI%20CONUS%20-%20%2010.35%20um%20-%20Band%2013&amp;time_drop=show
<br>https://www.star.nesdis.noaa.gov/GOES/conus_band.php?sat=G16&amp;band=13&amp;length=24
<br>
<br>AWIPS and the FAA weather system continue to exhibit the same issue as this 
link:  https://weather.cod.edu/satrad/
<br>
<br>(SEC) </span></div></div></td></tr></tbody></table></div></div>
<div><p>The following work notes were given about the Incident:</p><div 
style="margin-left:30px;"><table style="width:100%;"><tbody><tr><td><div 
style="padding:10px 15px; border:1px solid lightgray; border-left:7px solid 
gold;margin:2px 0;"><strong>2021-12-13 20:39:36 EST - Curtis Sims</strong><sup 
style="float:right;">Work notes</sup><div><span 
style="word-wrap:break-word;display:block;">@0114z - OMB received an update 
from  gilbert@xxxxxxxxxxxxxxxx
<br>reply from: gilbert@xxxxxxxxxxxxxxxx
<br>
<br>Someone from the College of DuPage reached out to me with an interesting 
observation from the University of Oklahoma...
<br>
<br>We had an outage on our NOAAport receive system this morning, which caused 
our systems to fail over to our partner organization's feeds. On investigation, 
we found that our receiver, ingest software, and LDM software were all working 
fine. We saw packets coming into the Linux-based ingest server, but nothing was 
being ingested. Then, it was noticed that the source IP of the NOAAport 
multicast traffic had changed, for at least some of the packets. Whereas in the 
past, it had been 10.0.9.51 , we are now seeing packets from 10.0.9.52 . Our 
ingest server was not accepting those because it had no special routing rule 
for that IP. We added such a rule (and did *not* delete the existing one for 
the .51 IP) and our NOAAport ingest began working as before.
<br>
<br>I can confirm the source IP for me is 10.0.9.52 as well, but unlike him 
that IP range was already open on my system and that traffic was making its way 
in. I'm not sure if this is related or not, but I wanted to pass along the 
intel in case it is.
<br>(CS)</span></div></div></td></tr><tr><td><div style="padding:10px 15px; 
border:1px solid lightgray; border-left:7px solid gold;margin:2px 
0;"><strong>2021-12-13 20:10:02 EST - Curtis Sims</strong><sup 
style="float:right;">Work notes</sup><div><span 
style="word-wrap:break-word;display:block;">@0054z - OMB received an update 
from Jeff Kelley
<br>reply from: jeff.kelley@xxxxxxxx
<br>
<br>I just got off the phone with NCF. They are aware that data dropouts are
<br>occurring (with all data types, not just satellite) after switching between
<br>systems earlier today. The tech mentioned that it might be a firewall
<br>issue.
<br>
<br>Jeff
<br>(CS)</span></div></div></td></tr></tbody></table></div></div>
<hr>
<div>&nbsp;</div><div style="display:inline">Ref:MSG5547071</div></body></html>

PNG image

  • 2021 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: