[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #HHK-349534]: LDM data feed down

Hi Wanhong,

re:
> Test...
> 
> 
> [ldm@weather3 ~]$ telnet idd.unidata.ucar.edu 388
> Trying 128.117.140.3...
> Connected to idd.unidata.ucar.edu.
> Escape character is '^]'.
> ?Connection closed by foreign host.

OK, good.

re:
> [ldm@weather3 ~]$ PuTTY
> -bash: PuTTY: command not found
> [ldm@weather3 ~]$
> [ldm@weather3 ~]$

re:
> [ldm@weather3 ~]$ notifyme -vl- -f ANY -h idd.tamu.edu
> Jul 24 19:49:52 notifyme[24781] NOTE: Starting Up: idd.tamu.edu: 
> 20170724194952.799 TS_ENDT {{ANY,  ".*"}}
> Jul 24 19:49:52 notifyme[24781] NOTE: LDM-5 desired product-class: 
> 20170724194952.799 TS_ENDT {{ANY,  ".*"}}
> Jul 24 19:49:52 notifyme[24781] INFO: Resolving idd.tamu.edu to 
> 128.194.76.175 took 0.08371 seconds
> Jul 24 19:49:53 notifyme[24781] ERROR: NOTIFYME(idd.tamu.edu): 7: Access 
> denied by remote server
> ^CJul 24 19:50:10 notifyme[24781] NOTE: exiting
> [ldm@weather3 ~]$ notifyme -vl- -f ANY -h idd.meteo.psu.edu
> Jul 24 19:50:13 notifyme[24782] NOTE: Starting Up: idd.meteo.psu.edu: 
> 20170724195013.087 TS_ENDT {{ANY,  ".*"}}
> Jul 24 19:50:13 notifyme[24782] NOTE: LDM-5 desired product-class: 
> 20170724195013.087 TS_ENDT {{ANY,  ".*"}}
> Jul 24 19:50:13 notifyme[24782] INFO: Resolving idd.meteo.psu.edu to 
> 192.5.158.38 took 0.094313 seconds
> Jul 24 19:51:03 notifyme[24782] NOTE: LDM-5 desired product-class: 
> 20170724195013.087 TS_ENDT {{ANY,  ".*"}}
> Jul 24 19:51:03 notifyme[24782] INFO: Resolving idd.meteo.psu.edu to 
> 192.5.158.38 took 0.001074 seconds
> Jul 24 19:51:03 notifyme[24782] ERROR: NOTIFYME(idd.meteo.psu.edu): 7: Access 
> denied by remote server
> ^CJul 24 19:51:08 notifyme[24782] NOTE: exiting

Given that Unidata (idd.unidata.ucar.edu), Penn State (idd.meteo.psu.edu) and 
Texas A&M
(idd.tamu.edu) are all configured to ALLOW reed REQUESTs from '.edu' machines, 
and
given that we see that one of the real server backends of our relay cluster,
idd.unidata.ucar.edu is denying a connection from 128.163.8.100 and we see no
connection attempts from any other machine with IP address that starts with
128.163, it is our best guess that the LDM feed REQUESTs from your two
machines are being proxyed by a machine with IP address 128.163.8.100.  This
would still work if 128.163.8.100 resolved to an '.edu' address, but it does 
not:

% nslookup 128.163.8.100
Server:         128.117.64.44
Address:        128.117.64.44#53

** server can't find 100.8.163.128.in-addr.arpa.: NXDOMAIN

Can you check with UKy network folks to see if they made a change
today sometime around 16:30 UTC?

> 
> 
> 
> ________________________________
> From: Unidata IDD Support <address@hidden>
> Sent: Monday, July 24, 2017 3:37:43 PM
> To: Priddy, Tom
> Cc: address@hidden; Priddy, Tom; Wang, Wanhong; Dixon, Matthew; Dixon, Matthew
> Subject: [LDM #HHK-349534]: LDM data feed down
> 
> Hi Wanhong,
> 
> re:
> > Tested on our servers: weather.uky.edu     and      weather3.ca.uky.edu
> >
> >
> >
> > [ldm@weather3 ~]$ notifyme -vl- -f "IDS|DDPLUS" -h idd.unidata.ucar.edu -o 
> > 3600
> > Jul 24 18:53:00 notifyme[24344] NOTE: Starting Up: idd.unidata.ucar.edu: 
> > 20170724175300.955 TS_ENDT {{IDS|DDPLUS, ".*"}}
> > Jul 24 18:53:00 notifyme[24344] NOTE: LDM-5 desired product-class: 
> > 20170724175300.955 TS_ENDT {{IDS|DDPLUS,  ".*"}}
> > Jul 24 18:53:00 notifyme[24344] INFO: Resolving idd.unidata.ucar.edu to 
> > 128.117.140.3 took 0.001386 seconds
> > Jul 24 18:53:01 notifyme[24344] ERROR: NOTIFYME(idd.unidata.ucar.edu): 7: 
> > Access denied by remote server
> > Jul 24 18:53:26 notifyme[24344] NOTE: LDM-5 desired product-class: 
> > 20170724175300.955 TS_ENDT {{IDS|DDPLUS,  ".*"}}
> > Jul 24 18:53:26 notifyme[24344] INFO: Resolving idd.unidata.ucar.edu to 
> > 128.117.140.3 took 0.000971 seconds
> > Jul 24 18:53:26 notifyme[24344] ERROR: NOTIFYME(idd.unidata.ucar.edu): 7: 
> > Access denied by remote server
> > ^CJul 24 18:53:36 notifyme[24344] NOTE: exiting
> 
> This is very strange since the LDM log files on all of the real server 
> backends
> of the idd.unidata.ucar.edu cluster have no entries for attempted 'notifyme'
> connections from any machine from the uky.edu domain, nor do they show that
> any 'notifyme' connections from machines from the 208.67 domain are being 
> denied.
> 
> What is the result of:
> 
> <as 'ldm'>
> telnet idd.unidata.ucar.edu 388
> 
> Also, what is the result of:
> 
> <as 'ldm'>
> notifyme -vl- -f ANY -h idd.tamu.edu
> 
> and:
> 
> notifyme -vl- -f ANY -h idd.meteo.psu.edu
> 
> re:
> > [ldm@weather3 ~]$ notifyme -vl- -f "IDS|DDPLUS" -h weather2.admin.niu.edu 
> > -o 3600
> > Jul 24 18:54:14 notifyme[24345] NOTE: Starting Up: weather2.admin.niu.edu: 
> > 20170724175414.370 TS_ENDT {{IDS|DDPLUS, ".*"}}
> > Jul 24 18:54:14 notifyme[24345] NOTE: LDM-5 desired product-class: 
> > 20170724175414.370 TS_ENDT {{IDS|DDPLUS,  ".*"}}
> > Jul 24 18:54:14 notifyme[24345] INFO: Resolving weather2.admin.niu.edu to 
> > 131.156.224.26 took 0.001468 seconds
> > Jul 24 18:55:04 notifyme[24345] NOTE: LDM-5 desired product-class: 
> > 20170724175414.370 TS_ENDT {{IDS|DDPLUS,  ".*"}}
> > Jul 24 18:55:04 notifyme[24345] INFO: Resolving weather2.admin.niu.edu to 
> > 131.156.224.26 took 0.000993 seconds
> > ^CJul 24 18:55:17 notifyme[24345] NOTE: exiting
> 
> The output from this 'notifyme' test and my test of the same here in Unidata 
> show
> that there is no LDM running on weather2.admin.niu.edu, or, if there is one
> running, it is not responding.
> 
> Cheers,
> 
> Tom
> --
> ****************************************************************************
> Unidata User Support                                    UCAR Unidata Program
> (303) 497-8642                                                 P.O. Box 3000
> address@hidden                                   Boulder, CO 80307
> ----------------------------------------------------------------------------
> Unidata HomePage                       http://www.unidata.ucar.edu
> ****************************************************************************
> 
> 
> Ticket Details
> ===================
> Ticket ID: HHK-349534
> Department: Support IDD
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> 
> 

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: HHK-349534
Department: Support IDD
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.