[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #PBI-640882]: rpc errors ?

Hi Donna,

If your public-facing LDM servers aren't missing any data, then what you're 
seeing is likely due to the two downstream LDM-s on the server deciding to 
switch their transfer modes from alternate to primary and vice versa. You 
should see log messages in the public LDM log files to this effect. Search for 
"switch" (case insensitive).

This automatic feature can lend itself to lots of log messages -- especially if 
the two upstream LDM-s are electronically close. Is this a problem?

> Sorry for the direct email. I would normally ping Gerry first but,
> according to FB, he and his family are in England ("across the pond").
> 
> I am seeing this in the ~ldm/var/logs/ldmd.log file of each of our .
> 
> Our set up has two relay-only LDM servers (ldm-relay0 and ldm-relay1) (they
> INGEST only) which have a private network address 10.0.30.26|27 as well as
> a public network address 165.91.55.26|27.
> 
> Then we have three "public" facing machines which provide data to
> downstream users and are managed through a load-balancer (ldm.tamu.edu).
> These three machines only ingest from ldm-relay0|1 (our LDM relays).
> 
> I may not be interpreting this correctly but, from the point of view of
> ldm3.tamu.edu (being a downstream "user" of ldm-relay0|1), it appears to me
> that the private network address is (repeatedly) being closed, for some
> reason, while data is still coming into the queue (I suppose from the
> public network address).
> 
> How can I find out what is going on here? Have you seen something like this
> before?
> 
> (in ldm3's ldmd.conf file:)
> > # ldm-relay0.tamu.edu
> > request         ANY     ".*"            10.0.30.26
> > request         ANY     ".*"            165.91.55.26
> > request         ANY     ".*"            ldm-relay0.tamu.edu
> >
> > # ldm-relay1.tamu.edu
> > request         ANY     "(.*)"          10.0.30.27
> > request         ANY     "(.*)"          165.91.55.27
> > request         ANY     "(.*)"          ldm-relay1.tamu.edu
> >
> 
> 
> The start up lines in ldmd.conf look normal:
> 
> > Mar 14 23:50:14 ldm3 ldmd[28179] NOTE: Starting Up (version: 6.12.14;
> > built: Mar 14 2016 23:34:03)?
> > Mar 14 23:50:14 ldm3 ldmd[28179] NOTE: Using local address 0.0.0.0:388
> > Mar 14 23:50:14 ldm3 rtstats[28181] NOTE: Starting Up (28179)
> > Mar 14 23:50:14 ldm3 rtstats[28182] NOTE: Starting Up (28179)
> > Mar 14 23:50:14 ldm3 10.0.30.26[28183] NOTE: Starting Up(6.12.14):
> > 10.0.30.26:388 20160315035014.883 TS_ENDT {{ANY, ".*"}}
> > Mar 14 23:50:14 ldm3 165.91.55.26[28184] NOTE: Starting Up(6.12.14):
> > 165.91.55.26:388 20160315035014.883 TS_ENDT {{ANY, ".*"}}
> > Mar 14 23:50:14 ldm3 ldm-relay0.tamu.edu[28185] NOTE: Starting
> > Up(6.12.14): ldm-relay0.tamu.edu:388 20160315035014.883 TS_ENDT {{ANY,
> > ".*"}}
> > Mar 14 23:50:14 ldm3 10.0.30.26[28183] NOTE: LDM-6 desired product-class:
> > 20160315035014.883 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=ca08a383eba2ce42a6912a61daeceab9"}}
> > Mar 14 23:50:14 ldm3 10.0.30.27[28186] NOTE: Starting Up(6.12.14):
> > 10.0.30.27:388 20160315035014.883 TS_ENDT {{ANY, "(.*)"}}
> > Mar 14 23:50:14 ldm3 165.91.55.27[28187] NOTE: Starting Up(6.12.14):
> > 165.91.55.27:388 20160315035014.884 TS_ENDT {{ANY, "(.*)"}}
> > Mar 14 23:50:14 ldm3 165.91.55.26[28184] NOTE: LDM-6 desired
> > product-class: 20160315035014.884 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=e9db74bbcf0476fdac76906c1f659d78"}}
> > Mar 14 23:50:14 ldm3 ldm-relay1.tamu.edu[28188] NOTE: Starting
> > Up(6.12.14): ldm-relay1.tamu.edu:388 20160315035014.884 TS_ENDT {{ANY,
> > "(.*)"}}
> > Mar 14 23:50:14 ldm3 ldm-relay0.tamu.edu[28185] NOTE: LDM-6 desired
> > product-class: 20160315035014.884 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=af63bcef23b29b7b515c5aeb45880574"}}
> > Mar 14 23:50:14 ldm3 10.0.30.27[28186] NOTE: LDM-6 desired product-class:
> > 20160315035014.884 TS_ENDT {{ANY, "(.*)"},{NONE,
> > "SIG=85b28eb791f293c829130ed81cc1928e"}}
> > Mar 14 23:50:14 ldm3 165.91.55.27[28187] NOTE: LDM-6 desired
> > product-class: 20160315035014.884 TS_ENDT {{ANY, "(.*)"},{NONE,
> > "SIG=2b1dff5e40fe17a37d2fb1f8b119eba3"}}
> > Mar 14 23:50:14 ldm3 ldm-relay1.tamu.edu[28188] NOTE: LDM-6 desired
> > product-class: 20160315035014.884 TS_ENDT {{ANY, "(.*)"},{NONE,
> > "SIG=84416ac0ec84f75437f3182c768c38da"}}
> > Mar 14 23:50:14 ldm3 10.0.30.26[28183] NOTE: Upstream LDM-6 on 10.0.30.26
> > is willing to be a primary feeder
> > Mar 14 23:50:14 ldm3 10.0.30.27[28186] NOTE: Upstream LDM-6 on 10.0.30.27
> > is willing to be a primary feeder
> > Mar 14 23:50:14 ldm3 165.91.55.26[28184] NOTE: Upstream LDM-6 on
> > 165.91.55.26 is willing to be an alternate feeder
> > Mar 14 23:50:14 ldm3 165.91.55.27[28187] NOTE: Upstream LDM-6 on
> > 165.91.55.27 is willing to be an alternate feeder
> >
> 
> ?<snip>?
> 
> Th
> ?en,
> repeat
> ?ing?
> ?
> ?every minute, there is an acknowledgement of ldm-relay0 (Primary), then
> "RPC layer closed connection" for the Private network address, then an
> acknowledgement of ldm-relay1 (Secondary), and again "RPC layer closed
> connection"
> for the Private network address; then a minute later: repeat
> ?s?
> :
> 
> Mar 15 10:27:41 ldm3 10.0.30.26[28183] NOTE: [ldm_config_file.c:803]
> > Switching data-product transfer-mode to alternate
> > Mar 15 10:27:41 ldm3 10.0.30.26[28183] NOTE: LDM-6 desired product-class:
> > 20160315142741.528 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=47176156fee26982d9bbf136049b668b"}}
> > Mar 15 10:27:41 ldm3 10.0.30.26[28183] NOTE: Upstream LDM-6 on 10.0.30.26
> > is willing to be an alternate feeder
> > Mar 15 10:28:20 ldm3 ldm-relay0.tamu.edu[28185] NOTE: LDM-6 desired
> > product-class: 20160315142820.971 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=3e9e4649dd983c051ae35ece949118f5"}}
> > Mar 15 10:28:20 ldm3 ldm-relay0.tamu.edu[28185] NOTE: Upstream LDM-6 on
> > ldm-relay0.tamu.edu is willing to be an alternate feeder
> > Mar 15 10:28:20 ldm3 165.91.55.26[28184] NOTE: [svc_tcp.c:358] EOF on
> > socket 3
> > Mar 15 10:28:20 ldm3 165.91.55.26[28184] NOTE: one_svc_run(): RPC layer
> > closed connection
> > Mar 15 10:28:20 ldm3 165.91.55.26[28184] NOTE: Connection to upstream LDM
> > closed: pid=26770
> > Mar 15 10:28:21 ldm3 ldm-relay1.tamu.edu[28188] NOTE: LDM-6 desired
> > product-class: 20160315142821.047 TS_ENDT {{ANY, "(.*)"},{NONE,
> > "SIG=aa79405c7d1946f1b3a287dcd677e457"}}
> > Mar 15 10:28:21 ldm3 ldm-relay1.tamu.edu[28188] NOTE: Upstream LDM-6 on
> > ldm-relay1.tamu.edu is willing to be an alternate feeder
> > Mar 15 10:28:21 ldm3 165.91.55.27[28187] NOTE: [svc_tcp.c:358] EOF on
> > socket 3
> > Mar 15 10:28:21 ldm3 165.91.55.27[28187] NOTE: one_svc_run(): RPC layer
> > closed connection
> > Mar 15 10:28:21 ldm3 165.91.55.27[28187] NOTE: Connection to upstream LDM
> > closed: pid=20005
> > Mar 15 10:29:20 ldm3 165.91.55.26[28184] NOTE: LDM-6 desired
> > product-class: 20160315142920.977 TS_ENDT {{ANY, ".*"},{NONE,
> > "SIG=531c1ca6d80819627a519bf229d6a181"}}
> >
> 
> ?I hope there is enough information here yet not too much. You can reach me
> by email or by text (or phone call) to 979.324.3549.
> 
> Thanks greatly, Donna

Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: PBI-640882
Department: Support LDM
Priority: Normal
Status: Closed