|
Derek,
Thanks for looking into the large lags. It has gotten to the point where our lags exceed 3600 seconds, and we start losing data.
I'm coming in from 144.92.131.136. My request statements look like:
# Split Feed
REQUEST CONDUIT "[09]$" conduit.ncep.noaa.gov
REQUEST CONDUIT "[18]$" conduit.ncep.noaa.gov
REQUEST CONDUIT "[27]$" conduit.ncep.noaa.gov
REQUEST CONDUIT "[36]$" conduit.ncep.noaa.gov
REQUEST CONDUIT "[45]$" conduit.ncep.noaa.gov
I can try with a single feed request too, if that might make any difference, or split it more than it is.
Might want to check with Tom Yoksas (cc'd) with Unidata to find out what IP they are feeding from. They are experiencing large lags as well.
Thanks,
Pete
--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
From: Derek VanPelt - NOAA Affiliate <address@hidden>
Sent: Thursday, February 21, 2019 12:25 PM
To: Pete Pokrandt
Cc: _NCEP.List.pmb-dataflow
Subject: Re: [Ncep.list.pmb-dataflow] Fwd: [conduit] Large lags on CONDUIT feed - started a week or so ago
HI Pete,
I am looking into the latencies you are seeing. Can you provide us with the ip you are coming in from so I can start tracking this down in our CONDUIT logs?
I am also investigating the health of the vms we are using.
Thank you,
Derek
Hi Pete,
We'll take a look and see if we can figure out what might be going on. We haven't done anything to try and address this yet, but based on your analysis I'm suspicious that it might be tied to a resource constraint on the VM or the blade it resides on.
Thanks,
Holly Uhlenhake
Acting Dataflow Team Lead
On Thu, Feb 21, 2019 at 11:32 AM Pete Pokrandt < address@hidden> wrote:
Just FYI, data is flowing, but the large lags continue.
Pete
--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
Data is flowing again - picked up somewhere in the GEFS. Maybe CONDUIT server was restarted, or ldm on it? Lags are large (3000s+) but dropping slowly
Pete
--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
Just a quick follow-up - we started falling far enough behind (3600+ sec) that we are losing data. We got short files starting at 174h into the GFS run, and only got (incomplete) data through 207h.
We have now not received any data on CONDUIT since 11:27 AM CST (1727 UTC) today (Wed Feb 20)
Pete
--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
Carissa,
We have been feeding CONDUIT using a 5 way split feed direct from
conduit.ncep.noaa.gov, and it had been really good for some time, lags 30-60 seconds or less.
However, the past week or so, we've been seeing some very large lags during each 6 hour model suite - Unidata is also seeing these - they are also feeding direct from
conduit.ncep.noaa.gov.
Any idea what's going on, or how we can find out?
Thanks!
Pete
--
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086 - address@hidden
_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web. Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.
conduit mailing list
address@hidden
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
Ncep.list.pmb-dataflow mailing list
address@hidden
https://www.lstsrv.ncep.noaa.gov/mailman/listinfo/ncep.list.pmb-dataflow
--
Derek Van Pelt
DataFlow Analyst
NOAA/NCEP/NCO
|
