Re: [ldm-users] GOES-16 reception issues when going through an enterprise level switch

  • To: "admin@xxxxxxxx" <admin@xxxxxxxx>
  • Subject: Re: [ldm-users] GOES-16 reception issues when going through an enterprise level switch
  • From: Gilbert Sebenste <gilbert@xxxxxxx>
  • Date: Tue, 14 Mar 2017 20:56:20 +0000
  • Authentication-results: ndws.com; dkim=none (message not signed) header.d=none;ndws.com; dmarc=none action=none header.from=niu.edu;
  • Spamdiagnosticmetadata: NSPM
  • Spamdiagnosticoutput: 1:99
Thanks, Ray! Isn't that just lovely...

Gilbert

> On Mar 14, 2017, at 3:55 PM, "admin@xxxxxxxx" <admin@xxxxxxxx> wrote:
> 
> This is common in enterprise switches for that filtering to be ON by default, 
> after some malware appeared that used multicast packets to do denial of 
> service.
> 
> Ray Weber
> 
> On Tuesday, March 14, 2017 4:50pm, "Gilbert Sebenste" <gilbert@xxxxxxx> said:
> 
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>> 
>> 
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/ Hey Gerry,
>> 
>> Itâs an Avaya 5510 (not 5520, sorry---the one has half of the number of
>> ports, but is otherwise the same as the 20).
>> By default, it rejects the flood of packets on those encapsulated IPâs. And
>> I now realize, COD isnât the only one having this issueâ
>> 
>> Gilbert
>> 
>> Gilbert Sebenste
>> Staff Meteorologist
>> 
>> Environmental Health and Safety
>> Labs for Wellness 154 | DeKalb, Illinois 60115
>> 815-753-5492
>> gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>
>> http://weather.admin.niu.edu<http://weather.admin.niu.edu/>
>> 
>> Everyone. Home. Safely.
>> 
>> [NIU]
>> 
>> 
>> From: Gerry Creager - NOAA Affiliate [mailto:gerry.creager@xxxxxxxx]
>> Sent: Tuesday, March 14, 2017 3:38 PM
>> To: Gilbert Sebenste <gilbert@xxxxxxx>
>> Cc: ldm-users@xxxxxxxxxxxxxxxx; NOAAPORT <noaaport@xxxxxxxxxxxxxxxx>
>> Subject: Re: [ldm-users] GOES-16 reception issues when going through an 
>> enterprise
>> level switch
>> 
>> Who'd have thought the switch wouldn't pass multicast? Is it possible the 
>> campus
>> network geeks told it not to? Using multicast for disseminating data like 
>> that
>> "just makes sense".
>> Glad you found THAT problem. What brand of switch, anyway? Cisco?
>> gerry
>> 
>> On Tue, Mar 14, 2017 at 3:30 PM, Gilbert Sebenste
>> <gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>> wrote:
>> Hello everyone,
>> 
>> The College of DuPage has a NOAAport receiver with a Novra S300N, and two 
>> Ubuntu
>> boxes as ingestors, so that if one fails, the other will still provide data. 
>> To
>> accomplish this, they use an enterprise level network switch, a Nortel/Avaya 
>> 5520,
>> that costs a lot of money (and not the $50 gigabit switches you can get from
>> Newegg, or wherever), to split the data feed so that both servers can get and
>> ultimately relay the data.
>> 
>> When trying to receive the test GOES-16 imagery months ago, it wasnât coming
>> across their NOAAport feed. I thought it was a glitch with their Novra box, 
>> or the
>> network switch. But now that GOES-16 data is being sent, neither NOAAPort 
>> ingest
>> server were getting the GOES-16 data on the NOTHER feed. What could be wrong?
>> 
>> We restarted the LDM on both servers, deleting and remaking the queues. That
>> didnât do anything.
>> We then double checked the ldmd.conf files, which had a few minor issues, 
>> and also
>> checked the logs,
>> of course. Still, no GOES-16 data was showing up.
>> We then did a packet sniff from one of the servers to the switch. No GOES-16 
>> test
>> data was being sent to the boxes from the switch!
>> 
>> We then made sure the Ubuntu kernel parameters were set correctly, per the 
>> LDM
>> instructions. One box didnât have it correct; the other was correct. We
>> rebooted both boxes after doing checks. That didnât get us GOES-16 data,
>> either.
>> 
>> So then, today, I convinced a College of DuPage meteorology program employee 
>> to go
>> out to the bitter cold dish farm and climate-controlled âdog houseâ
>> where the Novra and the servers are. As he power-cycled the Novra, I was 
>> doing an
>> ldmadmin watch on the NOTHER feed on the ingest servers. When the Novra came 
>> back
>> up after the power cyckeâno GOES-16 data was coming across to the servers!
>> 
>> So then we looked to the network switch. The employee grabbed a jumper 
>> cable, and
>> ran an Ethernet cable directly from the Novra to one of the two ingest 
>> servers.
>> Then, finally, GOES-16 data started pouring in on that server!
>> 
>> But why? How can a $1,000+ enterprise level switch have a problem like this?
>> 
>> Well, as it turns out, it wasnât the fault of the switch at all. See:
>> 224.0.1.[1-10] have reserved internet protocols on them:
>> http://www.networksorcery.com/enp/protocol/ip/multicast.htm
>> 
>> For those of you who donât like to understandably click on links from
>> strangers (and geeky meteorologists):
>> 
>>                                                                INTERNETWORK
>> CONTROL BLOCK
>> 
>> 224.0.1.0
>> 
>> VMTP<http://www.networksorcery.com/enp/protocol/vmtp.htm> Managers.
>> 
>> RFC 1045
>> 
>> 224.0.1.1
>> 
>> NTP<http://www.networksorcery.com/enp/protocol/ntp.htm>, Network Time 
>> Protocol.
>> 
>> RFC 1119<http://www.networksorcery.com/enp/rfc/rfc1119.pdf>
>> 
>> 224.0.1.2
>> 
>> SGI-Dogfight.
>> 
>> 
>> 
>> 224.0.1.3
>> 
>> Rwhod.
>> 
>> 
>> 
>> 224.0.1.4
>> 
>> VNP.
>> 
>> 
>> 
>> 224.0.1.5
>> 
>> Artificial Horizons - Aviator.
>> 
>> 
>> 
>> 224.0.1.6
>> 
>> NSS, Name Service Server.
>> 
>> 
>> 
>> 224.0.1.7
>> 
>> AUDIONEWS - Audio News Multicast.
>> 
>> 
>> 
>> 224.0.1.8
>> 
>> SUN NIS+ Information Service.
>> 
>> 
>> 
>> 224.0.1.9
>> 
>> MTP<http://www.networksorcery.com/enp/protocol/mtp.htm>, Multicast Transport
>> Protocol.
>> 
>> RFC 1301<http://www.networksorcery.com/enp/rfc/rfc1301.txt>
>> 
>> 224.0.1.10
>> 
>> IETF-1-LOW-AUDIO.
>> 
>> 
>> Uh huh. When the NWS picked those IP addresses to encapsulate the data, they
>> picked IP numbers that are used and reserved for enterprise-level switches.
>> 
>> As for the College of DuPage? I just got this email from head network 
>> engineer,
>> Dave Bukowski, who came up with this solution: you need to enable unknown
>> multicast flooding on those IPâs. Hereâs what Dave did, and his email
>> with the solution:
>> -----------
>> 
>> FOUND IT!!!!
>> 
>> 
>> This was in the config
>> 
>> show running-config
>> .....SNIP.....
>> !
>> ! *** IGMP ***
>> !
>> vlan igmp unknown-mcast-allow-flood 224.0.1.1
>> vlan igmp unknown-mcast-allow-flood 224.0.1.2
>> vlan igmp unknown-mcast-allow-flood 224.0.1.3
>> vlan igmp unknown-mcast-allow-flood 224.0.1.4
>> vlan igmp unknown-mcast-allow-flood 224.0.1.5
>> vlan igmp unknown-mcast-no-flood enable
>> .....ENDSNIP.....
>> 
>> 
>> Added
>> vlan igmp unknown-mcast-allow-flood 224.0.1.6
>> vlan igmp unknown-mcast-allow-flood 224.0.1.7
>> vlan igmp unknown-mcast-allow-flood 224.0.1.8
>> vlan igmp unknown-mcast-allow-flood 224.0.1.9
>> vlan igmp unknown-mcast-allow-flood 224.0.1.10
>> vlan igmp unknown-mcast-allow-flood 224.0.1.201
>> 
>> 
>> 
>> So now:
>> 
>> SDBDF01#show vlan igmp unknown-mcast-allow-flood
>> Allowed Multicast Addresses
>> -----------------------------------------------------------------------
>> 224.0.1.1       224.0.1.2       224.0.1.3       224.0.1.4
>> 224.0.1.5       224.0.1.6       224.0.1.7       224.0.1.8
>> 224.0.1.9       224.0.1.10     224.0.1.201
>> Total Multicast MAC Addresses: 0
>> Total Multicast IP  Addresses: 11
>> 
>> 
>> Now we have a flood of data coming in on both noaaport1 and noaaport2. It was
>> multicast and we were blocking the flood, but had to specifically allow the
>> flooding by multicast address.
>> 
>> -Dave
>> 
>> 
>> Thanks, Dave. We spent at least 10 hours trying to figure this out. On a $50
>> switch, you might not have a problem, but if you do decide to have multiple
>> ingesters by going through a network switchâif you donât get all of
>> the data, your reception may be fineâitâs the switch thatâs
>> been programmed to stop the flow of data.
>> 
>> Thanks to Tom Yoksas as he helped us try to figure this mess out, to Mike 
>> Zuranski
>> for going out in the bitter cold and spending hours on the phone as we tried 
>> to
>> figure this out, and for Dave Bukowski who ran the ball for the touchdown 
>> after we
>> discovered it was the switch blocking the data.
>> 
>> Gilbert
>> Gilbert Sebenste
>> Staff Meteorologist
>> 
>> Environmental Health and Safety
>> Labs for Wellness 154 | DeKalb, Illinois 60115
>> 815-753-5492<tel:(815)%20753-5492>
>> gilbert@xxxxxxx<mailto:gilbert@xxxxxxx>
>> http://weather.admin.niu.edu<http://weather.admin.niu.edu/>
>> Everyone. Home. Safely.
>> 
>> [NIU]
>> 
>> 
>> 
>> _______________________________________________
>> NOTE: All exchanges posted to Unidata maintained email lists are
>> recorded in the Unidata inquiry tracking system and made publicly
>> available through the web.  Users who post to any of the lists we
>> maintain are reminded to remove any personal information that they
>> do not want to be made public.
>> 
>> 
>> ldm-users mailing list
>> ldm-users@xxxxxxxxxxxxxxxx<mailto:ldm-users@xxxxxxxxxxxxxxxx>
>> For list information or to unsubscribe,  visit:
>> http://www.unidata.ucar.edu/mailing_lists/
>> 
>> 
>> 
>> --
>> Gerry Creager
>> NSSL/CIMMS
>> 405.325.6371
>> ++++++++++++++++++++++
>> âBig whorls have little whorls,
>> That feed on their velocity;
>> And little whorls have lesser whorls,
>> And so on to viscosity.â
>> Lewis Fry Richardson (1881-1953)
>> 
> 
>