[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050315: Large latency of met_research3.nchmf.gov.vn (cont.)

>From: Mai Nguyen <address@hidden>
>Organization: National Center For HydroMeteorological Forecasting, Hanoi, 
>Vietnam
>Keywords: 200503100444.j2A4inv2020321 IDD latency

Hi Mai,

>Thanks for your answers.

No worries.

>Yes, we've changed the order of priority in ldm.conf.
>The machine atm.geo.nsf.gov used to be our primary
>upstream host and idd.unidata.ucar.edu was the
>secondary. The 'miracle' happened after we swapped
>them. 

Sounds good.  Please check your ~ldm/logs/ldmd.log file to make sure
that your request to atm.geo.nsf.gov is getting through.  I suspect
that it is not since I do not see a connection from met_research3 on
atm.  Given this, I am guessing that the network connection from
met_research3 goes through the commodity Internet, not Internet2.  If
this is the case, your request will be blocked by the router that
controls port 388 traffic into/out of the NSF/ATM offices.  If requests
to atm are being blocked, it would be best to comment out them out of
your ldmd.conf file and stop and restart your LDM -- it will save
resources on your side.

I took the liberty of logging onto met_research3 to take a look at the
ldmd.log file and see that your feed requests to atm are, in fact, not
being serviced.  Here is a representative example of the messages I
saw:

Mar 15 04:30:36 met_research3 atm[28969]: ERROR: requester6.c:455; 
ldm_clnt.c:258: Couldn'
t connect to LDM 6 on atm.geo.nsf.gov 

This says that the request to atm.geo.nsf.gov is not being serviced.
Given this, it is best to comment out all ldmd.conf request lines to
atm and then stop and restart your LDM.  I did this for you; please
see below for additional information.

>Now, I have two more questions (not problems) to
>continue our communication ;-)

>1) Can we have more data feeds (apart from what we're
>having now i.e. HDS and IDD) for other kinds of data
>such as quickscat?

You can certainly get different kinds of data feeds if you want.
Quickscat data, however, is contained in the HDS stream under the
header ISXX.. .  There is a FILE action for the Quickscat data in the
GEMPAK Examples link off of the GEMPAK HomePage.

While on met_research3 I looked at the HDS request lines that were
setup and see that they do not include the Quickscat data:

request HDS     "(^H.[I-P]... KWB[^K])|(^H[A-Z][ABCD][A-Z][0-9][0-9] 
KWB.)|(.*(ECMWF|EGRR))"      idd.unidata.ucar.edu    PRIMARY

When I set these up, I limited the data being requested in the HDS feed
to global models output.  This pattern needs to be change to include
the Quickscat data or add another request entry.

The headers of the Quickscat products can be seen using the LDM utility
'notifyme'.  Here is an example with example output:

notifyme -vxl- -f HDS -o 10000 -h idd.unidata.ucar.edu -p ISXX

Mar 15 14:33:51 notifyme[20443]: Starting Up: idd.unidata.ucar.edu: 
20050315114711.204 TS_ENDT {{HDS,  "ISXX"}}
Mar 15 14:33:51 notifyme[20443]: Connected to upstream LDM-5
        NOTIFYME(idd.unidata.ucar.edu) returns OK
Mar 15 14:31:01 notifyme[20145]: c6bdba1ca4934b1b6a7b6133e1c037e9      137 
20050315140528.805     HDS 43063993  ISXX01 KNES 150903
Mar 15 14:31:01 notifyme[20145]: e08abeecc7688c15e6c742da19dc6887      165 
20050315140528.810     HDS 43063994  ISXX01 KNES 150904
Mar 15 14:31:01 notifyme[20145]: bd617d5c7483c558d4267c7dfad2b0ce      205 
20050315140530.950     HDS 43064027  ISXX01 KNES 150905
  ...


So, the data you want is available.  All that remains to be done is
request it.

I just took a quick look at the HDS latencies being seen on
met_research3.  The fact that they spike up to 1000 - 3000 seconds at
times tells me that the bandwidth to met_research3 is not very large,
OR that there is some sort of volume limiting being imposed on your
connection (we generically refer to a volume limiting as 'packet
shaping').  A comparison of the latencies for the low volume IDS|DDPLUS
feed versus the higher volume HDS feed is a classic signature of packet
shaping.  I recommend that you check with your network people and ask
them if there is any limiting being imposed on your IDD (port 388)
connections.

In the meantime, one way to get around packet shaping is to split up
feed requests into smaller pieces.  I just did this for you by
splitting the single HDS request to idd.unidata.ucar.edu into three
requests.  At the same time, I commented out the requests to
atm.geo.nsf.gov.

While editing ldmd.conf I notice that an entry to request the Quickscat
data was added by Tien on January 10, but it is commented out.  I think
that the reason it is commented out is it was not working, and I think
that the reason it was not working was that the pattern being requested
is incorrect.  I added a separate request for the Quickscat data, so
you should start getting it now.  You need to make sure that you have
the appropriate pqact.conf entry to process the data; please refer
to the GEMPAK Examples accessible through the GEMPAK HomePage.

>2) Is that possible (and neccesary) for us to have
>another machine (ie a new IP and DNS) to connect to
>IDD?

It is possible, but not necessary or advisable given the latency plots
I see for the HDS feed to met_research3 and the possibility that these
latencies are being caused by a lack of network bandwidth.

Instead, what I would do is setup a second machine and configure it to
ingest data from met_research3.  I would then setup execution of
'ldmfail' (a script contained in the LDM distribution that gets
installed in the ~ldm/bin directory) in the second machine's crontab to
monitor if the LDM on met_research3 is still working (ldmfail uses
ldmping to determine if the LDM on a host is working).  'ldmfail' will
switch ldmd.conf files and restart the LDM if/when it determines that
the LDM on the machine it ingesting from has stopped or is down.  The
idea behind 'ldmfail' is to have two ldmd.conf files: one contains
requests to one host, and the other to another host.  In your case,
one ldmd.conf file would be setup to request data only from met_research3
and the other would be setup to request data from idd.unidata.ucar.edu
in the exact same way that met_research3 does.

We can talk more about setting up an 'ldmfail' failover if you decide
to go ahead and configure a second machine to run the LDM.

>We would like to prepare for the quick
>replacement in the case if this machine dies.

I think that this is a wise move.

>Thank you very much as always.

No worries.

>And wish you a good working day!

You too.

Cheers,

Tom
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.