[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20051010: 20051006: 20051005: 20051003: pqact template for GFS conduit data

John,

Observing the CONDUIT latency at your psnldm site:
http://www.unidata.ucar.edu/cgi-bin/rtstats/iddstats_nc?CONDUIT+psnldm.nsbf.nasa.gov

Our previous assumption that the run started with little latency, but your site
continuously builds latency until you fall way behind is confirmed. This
could either be the bandwidth of your local network, or the limitation
of your firewall

The age of the oldest product available from idd.unidata.ucar.edu is
generally above 8000 seconds, so the 6000 second latency you have is still
receiving the products. About 1.8GB per run is being received by
psnldm which is the entire GFS grid #003 (f000-f180,  3 hourly, ~1.6+GB)
and GFS grid #002 (f192-f384, 12 hourly, ~80MB).

So, it appears that your psnldm machine can get the data now, although almost 2 
hours
late by the end of the cycle. This doesn't give any recoverability in the event 
of
network outage, but its a start. Hoefully you are seeing your local files 
reflect this
result.

Steve Chiswell
Unidata User SUpport

>From: "John Hobbie" <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200510110122.j9B1MMdm014877

>Hi Steve --
>
>Thanks for all your help on this.  
>
>I increased the size of the queue on psnldm, moved it to be the upstream node 
> so that it feeds newpsn.
>I also activated the rtstats on all three systems -- the one in Ft Sumner, too
>  -- so you should get the latency info as well.
>I changed all references to thelma to read idd.unidata.ucar.edu. 
>
>I am writing this on Monday, 10 Oct, while I am in San Francisco.  I will be o
> ut of pocket for the rest of the week, but I think I can tell if it is helpin
> g from back here by morning.
>
>Thanks again for all your help.
>
>Hobbie
>
>-----Original message-----
>From: Unidata Support address@hidden
>Date: Thu,  6 Oct 2005 17:15:36 -0500
>To: "John Hobbie" address@hidden
>Subject: 20051006: 20051005: 20051003: pqact template for GFS conduit data
>
>
>John,
>
>The problem you are describing sounds like your LDM relay is falling behind, a
> nd
>independent of any GEMPAK configuration.
>
>As an aside, you should be requesting data from idd.unidata.ucar.edu now
>since thelma.ucar.edu no longer exists, and is just a name alias. The 
>idd.unidata.ucar.edu is a cluster of hosts to provide you with the most
>reliability. Note that this is not related to your data stream problems.
>
>I am running a notifyme command to both your LDM machines to watch the CONDUIT
>feed arrive there. 
>
>Your old machine "newpsn" which is being fed from "thelma" is receiving the
>full gfs #003 per your description. The machine "psnldm" is feeding from "newp
> sn".
>The likely situation is that:
>
>The "newpsn" machine is falling way behing in receiving the CONDUIT feed.
>At this moment, it is receiving the F024 fields, and they are over 30 minutes
>behind. Once this machine falls more than 1 hour behind realtime in receiving 
> the data,
>your downstream machine "psnldm" will start missing data, since the default LD
> M configuration 
>is to ask for data up to 1 hour old, and on receipt of a product older than th
> at, it will ask to 
>skip ahead in time so as to catch up.....but since your upstream "newpsn" is t
> he one
>that is falling behind, "psnldm" is unable to catch up in time.
>
>If your machines were sending statistics in, using the rtstats program invocat
> ion from ldmd.conf,
>we would have a record of latency for your products and verify that the 1 hour
>  latency is occuring.
>I will use notifyme as a substitute for the time being and watch as the F084+ 
> times
>arrive, given you said that you see problems on your downstream in that time b
> etween F084
>and F180.
>
>The likely solution is to examine your LDM performance on "newpsn", the size o
> f your queue
>and its memory. If your plans are to replace that with the "psnldm", the best 
> case would
>be to have enough memory to hold your entire queue size, and your queue should
>  be
>large enough to hold at least 1 hours worth of data. The newer psnldm seems to
>be keeping up with "newpsn" in terms of feed volume, so we should see data dro
> ps as the
>upstream falls over 1 hour behind, and prpbably RECLASS messages in your ldmd.
> log file.
>
>Steve Chiswell
>Unidata User Support
>
>Steve Chiswell
>Unidata User Support
>
>
>
>
>>From: "John Hobbie" <address@hidden>
>>Organization: UCAR/Unidata
>>Keywords: 200510062146.j96LkLG7002275
>
>>Steve --
>>
>>More info on this problem.  I hope this might give some insight into what is 
> h
>> appening.
>>
>>Today, I sat with GARP running on both machines watching GFS003 come in -- th
> e
>>  12Z data set.  The data showed up similtaneously on both machines up throug
> h
>>  84 hours.  The older machine, "newpsn", which is fed by thelma, kept on fil
> l
>> ing out GFS003 past 84 hours through 180 hours.  The machine with the latest
>  
>> GEMPAK/LDM versions, "psnldm" which is fed by "newpsn", only got 9208 fields
> ,
>>  while the "newpsn" machine captured 20026 fields.
>>
>>The other problem is that number of fields logged in GFS003 on "psnldm" varie
> s
>>  from run to run, but hovers around 9500 fields.   There do not appear to be
>  
>> any missing fields within the data that is captured (skimming the results of
>  
>> GDINFO.)  That is, even though psnldm is not capturing the entire length of 
> t
>> he set (180 hours), it captures every field until it stops about half way th
> r
>> ough the data (about 84-90 hours).
>>
>>I don't know what is filling up -- number of files open or memory overflow, e
> t
>> c. -- but the same general problem arises whether day or night or product ru
> n
>>  (18Z or 06Z).
>>
>>
>>Do you have any ideas what these symptoms are indicating?  
>>
>>
>>Hobbie 
>>
>--
>NOTE: All email exchanges with Unidata User Support are recorded in the
>Unidata inquiry tracking system and then made publicly available
>through the web.  If you do not want to have your interactions made
>available in this way, you must let us know in each email you send to us.
>
--
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.