Justin,
Is that box capable of sending stats to our rtstats.unidata.ucar.edu
host?
Eg, is it allowed to connect outside your domain?
The ldm won't need to run pqact to test out the throughput and
netwrok,
but will need ldmd.conf lines:
EXEC "rtstats -h rtstats.unidata.ucar.edu"
request CONDUIT ".*" ncepldm.woc.noaa.gov
The pqact EXEC action can be commented out. The request
line will start the feed to ncepldm which flood.atmos.uiuc.edu is
pointing to, and showing high latency. If you are able to feed from
ncepldm
without the latency that outside hosts are showing, then it would
isolate the
problem further to the border of your network to the outside. If
you do
show similar latency, then it would either be the LDM configuration
itself, or the local
router that the machines are on.
If you are able to send rtstats out to us, then we can monitor
stats on
our web pages.
Your network might require a static route be added for sending that
outside your domain (that would something your networking folks
would
know). The rtstats sends
a small text report about every 60 seconds, so not a lot of traffic.
If you can't configure your host to send rtstats, then we could
create
q
pqact.conf action to file the .status reports and calculate the
latency
from those.
Thanks,
Steve
On Wed, 2007-06-20 at 12:03 -0400, Justin Cooke wrote:
Steve,
If you provide us a pqact.conf I can have the box chi set up to
feed
off of ldm1 and see how its latencies are.
Justin
On Jun 20, 2007, at 11:36 AM, Steve Chiswell wrote:
Justin,
Since the change at 13Z by dropping daffy.unidata.ucar.edu out
of the
top level nodes the ldm2 feed to NSF is showing little/no
latency at
all. The ldm1 feed to NSF which is connected using the
alternate LDM
mode is only devivering the .status messages its creates as all
the
other products are duplicates of products already being
received from
LDM2 and that is showing high latency:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?
CONDUIT+atm.cise-nsf.gov
This configuration is getting data out to the community at the
moment.
The downside here is that it puts a single point of failure at
NSF in
getting the data to Unidata, but
I'll monitor that end.
It seems that ldm1 is either slow, or it is showing network
limitations
(since
flood.atmos.uiuc.edu is feeding from ncepldm which is apparently
pointing to ldm1, there is load on ldm1 besides the NSF feed.
LDM2 is
feeding both NSF and idd.aos.wisc.edu (and Wisc looks good
since 13Z
as
well) so it is able to
handle the throughput to 2 downstreams, but adding daffy as the
3rd
seems to
cross some point in volume of what can be sent out.
Steve
On Wed, 2007-06-20 at 09:45 -0400, Justin Cooke wrote:
Thanks Steve,
Chi has set up a box on the lan for us to run LDM on, I am
beginning
to
get things running on there.
have you seen any improvement since dropping daffy?
Justin
On Jun 20, 2007, at 9:03 AM, Steve Chiswell wrote:
Justin,
Yes, this does appear to be the case. I will drop daffy from
feeding
directly and instead move it to feed from NSF. That will
remove one
of the top level relays of data having to go out of NCEP and
we can see if the other nodes show an improvement.
Steve
On Wed, 20 Jun 2007, Justin Cooke wrote:
Steve,
Did you see a slowdown to ldm2 after Pete and the other sites
began
making connections?
Chi, considering steve saw a good connection to ldm1 before the
other
sites connected doesn't that point toward a network issue?
All of our queue processing on the diskserver has been running
without
any problems so I don't believe anything on that system would
impacting
ldm1/ldm2.
Justin
On Jun 20, 2007, at 12:04 AM, Chi Y Kang wrote:
I setup the test LDM server for the NCEP folks to test the
local
pull
from the LDM servers. That should give us some information /
network
or system related issue. We'll handle that tomorrow. I am a
little
bit concerned that the slow down all occurred at the some
time as
the
ldm1 crash last week.
Also, can NCEP also check if there are any bad dbnet queues on
the
backend servers? Just to verify.
Steve Chiswell wrote:
Thanks Justin,
I also had a typo in my message:
ldm1 is running slower than ldm2
Now if the feed to ldm2 all of a sudden slows down if Pete
and
other
sites add a request to it, it would really signal some
sort of
total
bandwidth limitation
on the I2 connection. Seemed a little coincidental that we
had a
show
period
of good connectivity to ldm1 after which it slowed way down.
Steve
On Tue, 2007-06-19 at 17:01 -0400, Justin Cooke wrote:
I just realized the issue. When I disabled the "pqact"
process
on
ldm2 earlier today it caused our monitor script (in cron,
every 5
min) to kill the LDM and restart it. I have removed the
check
for
the pqact in that monitor...things should be a bit better
now.
Chi.Y.Kang wrote:
Huh, i thought you guys were on the system. let me take a
look
on
ldm2
and see what is going on.
Justin Cooke wrote:
Chi.Y.Kang wrote:
Steve Chiswell wrote:
Pete and David,
I changed the CONDUIT request lines at NSF and
Unidata to
request data
from ldm1.woc.noaa.gov rather than ncepldm.woc.noaa.gov
after
seeing
lots of
disconnect/reconnects to the ncepldm virtual name.
The LDM appears to have caught up here as an interim
solution.
Still don't know the cause of the problem.
Steve
I know the NCEP was stop and starting the LDM service
on the
ldm2
box
where the VIp address is pointed to at this time. how is
the
current
connection to LDM1? is the speed of the conduit feed
acceptable?
Chi, NCEP has not restarted the LDM on ldm2 at all
today. But
looking
at the logs it appears to be dying and getting
restarted by
cron.
I will watch and see if I see anything.
Justin
--
Chi Y. Kang
Contractor
Principal Engineer
Phone: 301-713-3333 x201
Cell: 240-338-1059
--
Steve Chiswell <address@hidden>
Unidata
--
Steve Chiswell <address@hidden>
Unidata
