[ldm-users] GOES-16 reception issues when going through an enterprise level switch

  • To: "ldm-users@xxxxxxxxxxxxxxxx" <ldm-users@xxxxxxxxxxxxxxxx>, NOAAPORT <noaaport@xxxxxxxxxxxxxxxx>
  • Subject: [ldm-users] GOES-16 reception issues when going through an enterprise level switch
  • From: Gilbert Sebenste <gilbert@xxxxxxx>
  • Date: Tue, 14 Mar 2017 20:30:29 +0000
  • Authentication-results: unidata.ucar.edu; dkim=none (message not signed) header.d=none; unidata.ucar.edu; dmarc=none action=none header.from=niu.edu;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99
Hello everyone,

The College of DuPage has a NOAAport receiver with a Novra S300N, and two 
Ubuntu boxes as ingestors, so that if one fails, the other will still provide 
data. To accomplish this, they use an enterprise level network switch, a 
Nortel/Avaya 5520, that costs a lot of money (and not the $50 gigabit switches 
you can get from Newegg, or wherever), to split the data feed so that both 
servers can get and ultimately relay the data.

When trying to receive the test GOES-16 imagery months ago, it wasn't coming 
across their NOAAport feed. I thought it was a glitch with their Novra box, or 
the network switch. But now that GOES-16 data is being sent, neither NOAAPort 
ingest server were getting the GOES-16 data on the NOTHER feed. What could be 
wrong?

We restarted the LDM on both servers, deleting and remaking the queues. That 
didn't do anything.
We then double checked the ldmd.conf files, which had a few minor issues, and 
also checked the logs,
of course. Still, no GOES-16 data was showing up.
We then did a packet sniff from one of the servers to the switch. No GOES-16 
test data was being sent to the boxes from the switch!

We then made sure the Ubuntu kernel parameters were set correctly, per the LDM 
instructions. One box didn't have it correct; the other was correct. We 
rebooted both boxes after doing checks. That didn't get us GOES-16 data, either.

So then, today, I convinced a College of DuPage meteorology program employee to 
go out to the bitter cold dish farm and climate-controlled "dog house" where 
the Novra and the servers are. As he power-cycled the Novra, I was doing an 
ldmadmin watch on the NOTHER feed on the ingest servers. When the Novra came 
back up after the power cycke...no GOES-16 data was coming across to the 
servers!

So then we looked to the network switch. The employee grabbed a jumper cable, 
and ran an Ethernet cable directly from the Novra to one of the two ingest 
servers. Then, finally, GOES-16 data started pouring in on that server!

But why? How can a $1,000+ enterprise level switch have a problem like this?

Well, as it turns out, it wasn't the fault of the switch at all. See: 
224.0.1.[1-10] have reserved internet protocols on them:

http://www.networksorcery.com/enp/protocol/ip/multicast.htm

For those of you who don't like to understandably click on links from strangers 
(and geeky meteorologists):

                                                                INTERNETWORK 
CONTROL BLOCK

224.0.1.0

VMTP<http://www.networksorcery.com/enp/protocol/vmtp.htm> Managers.

RFC 1045

224.0.1.1

NTP<http://www.networksorcery.com/enp/protocol/ntp.htm>, Network Time Protocol.

RFC 1119<http://www.networksorcery.com/enp/rfc/rfc1119.pdf>

224.0.1.2

SGI-Dogfight.



224.0.1.3

Rwhod.



224.0.1.4

VNP.



224.0.1.5

Artificial Horizons - Aviator.



224.0.1.6

NSS, Name Service Server.



224.0.1.7

AUDIONEWS - Audio News Multicast.



224.0.1.8

SUN NIS+ Information Service.



224.0.1.9

MTP<http://www.networksorcery.com/enp/protocol/mtp.htm>, Multicast Transport 
Protocol.

RFC 1301<http://www.networksorcery.com/enp/rfc/rfc1301.txt>

224.0.1.10

IETF-1-LOW-AUDIO.


Uh huh. When the NWS picked those IP addresses to encapsulate the data, they 
picked IP numbers that are used and reserved for enterprise-level switches.

As for the College of DuPage? I just got this email from head network engineer, 
Dave Bukowski, who came up with this solution: you need to enable unknown 
multicast flooding on those IP's. Here's what Dave did, and his email with the 
solution:

-----------

FOUND IT!!!!


This was in the config

show running-config
.....SNIP.....
!
! *** IGMP ***
!
vlan igmp unknown-mcast-allow-flood 224.0.1.1
vlan igmp unknown-mcast-allow-flood 224.0.1.2
vlan igmp unknown-mcast-allow-flood 224.0.1.3
vlan igmp unknown-mcast-allow-flood 224.0.1.4
vlan igmp unknown-mcast-allow-flood 224.0.1.5
vlan igmp unknown-mcast-no-flood enable
.....ENDSNIP.....


Added
vlan igmp unknown-mcast-allow-flood 224.0.1.6
vlan igmp unknown-mcast-allow-flood 224.0.1.7
vlan igmp unknown-mcast-allow-flood 224.0.1.8
vlan igmp unknown-mcast-allow-flood 224.0.1.9
vlan igmp unknown-mcast-allow-flood 224.0.1.10
vlan igmp unknown-mcast-allow-flood 224.0.1.201



So now:

SDBDF01#show vlan igmp unknown-mcast-allow-flood
Allowed Multicast Addresses
-----------------------------------------------------------------------
224.0.1.1       224.0.1.2       224.0.1.3       224.0.1.4
224.0.1.5       224.0.1.6       224.0.1.7       224.0.1.8
224.0.1.9       224.0.1.10     224.0.1.201
Total Multicast MAC Addresses: 0
Total Multicast IP  Addresses: 11


Now we have a flood of data coming in on both noaaport1 and noaaport2. It was 
multicast and we were blocking the flood, but had to specifically allow the 
flooding by multicast address.

-Dave



Thanks, Dave. We spent at least 10 hours trying to figure this out. On a $50 
switch, you might not have a problem, but if you do decide to have multiple 
ingesters by going through a network switch...if you don't get all of the data, 
your reception may be fine...it's the switch that's been programmed to stop the 
flow of data.

Thanks to Tom Yoksas as he helped us try to figure this mess out, to Mike 
Zuranski for going out in the bitter cold and spending hours on the phone as we 
tried to figure this out, and for Dave Bukowski who ran the ball for the 
touchdown after we discovered it was the switch blocking the data.

Gilbert

Gilbert Sebenste
Staff Meteorologist

Environmental Health and Safety
Labs for Wellness 154 | DeKalb, Illinois 60115
815-753-5492
gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>
http://weather.admin.niu.edu<http://weather.admin.niu.edu/>

Everyone. Home. Safely.

[NIU]


PNG image

  • 2017 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: