[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[NOAAPORT #BGK-918302]: NOAAPort ingest not working

Hi,

re:
> Thank you very much for your quick response. We couldn't have configured ldm 
> without
> your help. Your help is very much appreciated!

No worries.

re:
> Giving you a little background on our NOAA port system (S300), it is 
> connected to a
> Linksys 5-port network hub, then our legacy server (which is up and running 
> ldm and NOAA
> port for our production environment) and two of our new servers (a QA and 
> PRD, which we
> are configuring them to replace the legacy server) are connected to the 
> Linksys hub
> (which is a dummy hub that doesn’t block any multicast traffic).

We do pretty much the same thing here: the output from each of our 2 Novra 
S300Ns is
connected to a high quality hub so that multiple machines running NOAAPort 
ingest
can be fed (we have three machines we use for testing and 2 that are feeding 
into
the IDD).

re:
> The legacy server is
> running Red hat Linux enterprise server version 4.1 (that is why we are 
> decommissioning
> it) and ldm version 6.9.8.

Ouch!

re:
> The new servers are running red hat enterprise 7.6 and ldm
> 6.13.7.

The latest LDM release is v6.13.11.  I advise you to upgrade as soon as 
practical
(the version, however, has nothing to do with the problem you are 
experiencing!).

re:
> Before answering your questions I also want to mention that our goal is to 
> get 8
> of the NOAA port channels (we are getting only 4 in the legacy servers, 
> adding GOES-R for
> example wouldn't work and I think it could be related to the network hub, but 
> that is
> another question we can address when we get NOAA port working).

OK.

re:
> Coming back to your questions,
> 
> 
> >>Our recommendation is to set the following in /etc/sysctl.conf:
> 
> # Disable IPv6
> >>net.ipv6.conf.all.disable_ipv6=1
> >>net.ipv6.conf.default.disable_ipv6=1
> >>#
> >># Enable DVBS reception
> >>#
> >>net.ipv4.conf.default.rp_filter = 2
> >>#
> >># DVBS multicast fragment reassembly
> >>#
> >>net.ipv4.ipfrag_max_dist = 0

> I will send the contents of the file in a follow up email. I
> remember I set net.ipv4.ipfrag_max_dist=4094 to match the working
> legacy server (which has a different version of red hat and ldm though).

We have used the /etc/sysctl.conf settings pretty much from the first
day we started ingesting the DVBS-2 version of NOAAPort.

re:
> I will verify the other config values and send them
> in a follow up email.

OK.

re:
> >>Questions:
> 
> >>- what OS/version are you using?
> 
> AME="Red Hat Enterprise Linux Server"
> VERSION="7.6 (Maipo)"

OK.  This is the same version (but CentOS, not RHE) that the site I referred
to was trying to get working.

re:
> >>- what version of 'tcpdump' are you using?
> tcpdump version 4.9.2

OK.

re: did you make sure that the firewall on your machine was configured to
allow all traffic from the Ethernet interface that your Novra S300N

> 
> Yes, it is Novra S300 and as I described it above it is connected to a 
> Linksys 5 port
> hub, and a legacy server connected to one of the ports and running ldm and 
> working
> noaaport. The new servers are connected to the same hub.

As I noted above, this is pretty much the same way that we are configured here.

re:
> For the none NOAAport ldm data,
> we requested vendors to add our new servers to their firewall permitted list.

I do not understand this comment.  The Ethernet output from a Novra S300N is 
connected
to an private Ethernet interface on a machine that will run the LDM and NOAAPort
ingest.  The firewall on this machine needs to be configured to allow all of the
traffic on the Ethernet interface.  Many sites have gotten to the point you are
now - tcpdump shows traffic arriving at the Ethernet interface - only to be
thwarted by the firewall on their machine not being configured to allow the
traffic.  In cases where multicast routing is properly configured and the LDM
is built and installed correctly, re-configuring the firewall to allow all 
traffic on the Ethernet interface has resulted in the NOAAPort component of
the LDM build to start ingesting.

re:
> The traffic
> from noaaport interface is permitted.

Through the machine's firewall?

re:
> I was listing the traffic using tcpdump and that is
> when I saw the bad packet size message in all the stream.

tcpdump looks at the Ethernet interface directly, it sees the
traffic that is arriving on the upstream side of the firewall.

re: what does your routing look like

> Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
> 0.0.0.0         10.246.188.1    0.0.0.0         UG        0 0          0 bond0
> 10.1.1.0        0.0.0.0         255.255.255.0   U         0 0          0 em4
> 10.246.188.0    0.0.0.0         255.255.255.0   U         0 0          0 bond0
> 169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 em4
> 169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 bond0
> 224.0.1.1       0.0.0.0         255.255.255.255 UH        0 0          0 em4
> 224.0.1.2       0.0.0.0         255.255.255.255 UH        0 0          0 em4
> 224.0.1.3       0.0.0.0         255.255.255.255 UH        0 0          0 em4
> 224.0.1.4       0.0.0.0         255.255.255.255 UH        0 0          0 em4

Hmm... What is 'bond0'?  Do you only have one Ethernet interface on the machine
this listing was done on?

For comparison, here is what our machines look like:

~: netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
128.117.156.0   0.0.0.0         255.255.255.0   U         0 0          0 eth0
192.168.1.0     0.0.0.0         255.255.255.0   U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth1
169.254.0.0     0.0.0.0         255.255.0.0     U         0 0          0 eth0
224.0.0.0       192.168.1.7     240.0.0.0       UG        0 0          0 eth1
0.0.0.0         128.117.156.251 0.0.0.0         UG        0 0          0 eth0

Notice that we connect the Ethernet from the Novra to a separate Ethernet
interface, eth1, and the machine is connected to the net via eth0.

re: what are the NOAAPort exec lines in your LDM configuration file?
Ours look like:

> >># 20170313 - changed set of noaaportIngester instances to match:
> >>#
> >>#            CHANNEL PID MULTICAST ADDRESS Port DETAILS
> >>#            NMC     101     224.0.1.1     1201 NCEP / NWSTG
> >>#            GOES    102     224.0.1.2     1202 GOES / NESDIS
> >>#            NMC2    103     224.0.1.3     1203 NCEP / NWSTG2
> >>#            NOPT    104     224.0.1.4     1204 Optional Data - OCONUS 
> >>Imagery / Model
> >>#            NPP     105     224.0.1.5     1205 National Polar-Orbiting 
> >>Partnership / POLARSAT
> >>#            EXP     106     224.0.1.8     1208 Experimental
> >>#            GRW     107     224.0.1.9     1209 GOES-R Series West
> >>#            GRE     108     224.0.1.10    1210 GOES-R Series East
> >>#            NWWS    201     224.1.1.1     1201 Weather Wire
> >>#
> >>exec    "noaaportIngester -n -m 224.0.1.1  -l /data/tmp/nwstg.log"
> >>exec    "noaaportIngester -n -m 224.0.1.2  -l /data/tmp/goes.log"
> >>exec    "noaaportIngester -n -m 224.0.1.3  -l /data/tmp/nwstg2.log"
> >>exec    "noaaportIngester -n -m 224.0.1.4  -l /data/tmp/oconus.log"
> >>exec    "noaaportIngester -n -m 224.0.1.5  -l /data/tmp/nother.log"
> >>exec    "noaaportIngester -n -m 224.0.1.8  -l /data/tmp/nother.log"
> >>exec    "noaaportIngester -n -m 224.0.1.9  -l /data/tmp/nother.log"
> >>exec    "noaaportIngester -n -m 224.0.1.10 -l /data/tmp/nother.log"

Note that we are using the combined NOAAPort ingest module, noaaportIngester.
It is our recommendation that you do the same on your new machine.

re:
> The following are the dvbs and noaaport entries on ldmd.conf (again we just 
> matched them
> with config on the legacy, we wanted to have GOES-R on legacy but it didn’t 
> work [we
> think it is related to the dummy linksys hub])
> 
> # dvbs shared memory ingest processes
> exec    "dvbs_multicast -v -m 224.0.1.1"
> exec    "dvbs_multicast -v -m 224.0.1.2"
> exec    "dvbs_multicast -v -m 224.0.1.3"
> exec    "dvbs_multicast -v -m 224.0.1.4"
> #
> # readnoaaport shared memory readers
> exec    "readnoaaport -v  -m 224.0.1.1"
> exec    "readnoaaport -v  -m 224.0.1.2"
> exec    "readnoaaport -v  -m 224.0.1.3"
> exec    "readnoaaport -v  -m 224.0.1.4"

This is the old setup.  I can't recall if the NOAAPort ingest code was
bundled with the version of the LDM that you are using on your legacy
machine.  New versions of the LDM have the NOAAPort ingest code
as part of the distribution.

The other thing that has been changed in new releases of the LDM is
logging.  We have moved away from 'syslog' logging to use of a
logging package that is part of the LDM.  This gets aware from the
need to configure 'syslog' logging for LDM use, a big win.

re: are the permissions on LDM executables correct?
> 
> >> There are 4 routines in an LDM build that must have setuid root
> >> privilege:
> 
> >>-rwsr-xr-- 1 root ustaff  60025 May 14 19:10 ldmd
> >>-rwsr-xr-- 1 root ustaff  11460 May 14 19:10 hupsyslog
> >>-rwsr-xr-- 1 root ustaff 407007 May 14 19:10 noaaportIngester
> >>-rwsr-xr-- 1 root ustaff 324768 May 14 19:10 dvbs_multicast
> 
> >> Does your build have the correct permissions for these routines?
> >> Please send the output of:
> 
> >><as 'ldm'>
> >>cd ~ldm
> >>ls -alt bin/
> 
> Here is the output of the command you mentioned. The permissions are set 
> correctly.
> -rwxr-xr-x. 1 ldm  ldm 395800 Mar  4 16:16 readnoaaport
> -rwsr-xr--. 1 root ldm  69936 Feb  7 16:50 ldmd
> -rwsr-xr--. 1 root ldm  12808 Feb  7 16:50 hupsyslog
> -rwsr-xr--. 1 root ldm 464384 Feb  7 16:50 noaaportIngester
> -rwsr-xr--. 1 root ldm 367832 Feb  7 16:50 dvbs_multicast

Yes, the permissions are set correctly - setuid root for ldmd, hupsyslog,
noaaportIngester and dvbs_multicast.  I still strongly recommend to switching
to use of the combined ingest module, noaaportIngester.

Remember, when you make any change in the LDM configuration file, you
will need to stop and then start the LDM for the change(s) to take
effect.

The things that are bothersome to me are:

- your multicast routing setup

- the possibility that your NOAAPort ingest and network connection are
  using the same Ethernet interface

Here is the URL for a web page that talks about the setup under CentOS 7:

https://ahelpme.com/linux/multicast/receive-multicast-packets-on-centos-7-and-other-linux-distros/

Question:

Is there any chance that we could login to your machine to take a look?

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: BGK-918302
Department: Support NOAAPORT
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.