Re: [conduit] Recent CONDUIT latencies?

• To: "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>, "support-conduit@xxxxxxxxxxxxxxxx" <conduit@xxxxxxxxxxxxxxxx>, Anne Myckow - NOAA Federal <anne.myckow@xxxxxxxx>, Pete Pokrandt <poker@xxxxxxxxxxxx>
• Subject: Re: [conduit] Recent CONDUIT latencies?
• From: "Person, Arthur A." <aap1@xxxxxxx>
• Date: Mon, 17 May 2021 17:14:06 +0000
• Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=psu.edu; dmarc=pass action=none header.from=psu.edu; dkim=pass header.d=psu.edu; arc=none
• Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=M1h5geIUBL38ktIxvBip/n/IAhFwuPPprF8DTftaEZU=; b=YZpOBhELEx0TAkldhzjoeQoKWMURUJruMUAThgBsYinzQrhxTrhLcQAf5dVlddGKIH0Ac35RU6xnVObdkwBX6l7QMiP9hBpOqDiP+AI/kiga4p4cREex7/IFhsPEjSm+NdfFrxNSsJ6+3TNubwaTje/dl3atLNsSs1JDBEaO0Zbz2J7csLQeVzhvWDx4IlIdTo7HPKIBRYb1vHdGqDqvXCHa6oFSM8/9EROeEjLNiZF5It3cpmQR2QcWgqmze+kgeOn78roYk11HG0hwzoXG8MHg5p+13cfyNqL1RD3l5sgKiEEkwjaUKcD9zMznxHV5aBmkb+kUNozvkA1dAWdj+g==
• Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=ODVDRC0cwVakZrDzEaldejo5gS6UsWHK5spainMQsvAwHAPjt4C3+0yFdXLZyFFyhSite3Wsmdv74D+9BP0qu75vS4Gm2WZJXkKL9MuIxqQTY7S3A/3zHNaQ28/jBeh0rLJJHVvQYwSj5MJAJh0TITwzssWKQGXj2WdulK1Bwk7FLpEzZK8vxrba9WqjOM+8F3nsKY8eDRz11s6c/TmK06Tkcg2shROdg71ZKi1kuRan/rmAd2p/R9lWYtzbtWO3VDpPrwLFtkE8MnkeKv8xDPB4BlI+LxwmCiZ2trioqMK9WcXziSP7noLKfQCq7Ow8io+7WTS2U+kZNEOAbHv3fQ==
• Authentication-results: iastate.edu; dkim=none (message not signed) header.d=none; iastate.edu; dmarc=none action=none header.from=psu.edu;

Anne,

Our latencies, presently, are pretty normal, as Pete said, around 30-60 seconds:

[cid:b04cdaf7-15bb-4a08-b47c-91fc8ceee936]

Right now, we're running a 20-way split on the requests.  Regarding latencies, 
they can be affected not only by network parameters, but also by local system 
parameters, particularly I/O on local devices where a queue might reside.

                    Art


Arthur A. Person
Assistant Research Professor, System Administrator
Penn State Department of Meteorology and Atmospheric Science
email:  aap1@xxxxxxx, phone:  814-863-1563<callto:814-863-1563>

________________________________
From: conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Herzmann, Daryl E 
[AGRON] <akrherz@xxxxxxxxxxx>
Sent: Monday, May 17, 2021 11:51 AM
To: support-conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>; Anne Myckow - 
NOAA Federal <anne.myckow@xxxxxxxx>; Pete Pokrandt <poker@xxxxxxxxxxxx>
Subject: Re: [conduit] Recent CONDUIT latencies?

Howdy Pete,

How heavily sharded (how many LDM FEED requests) are you making to get all the 
CONDUIT data that you need?  If you totaled up all your feed request volume and 
divided by the number of LDM requests, what rough number do you get for 
bandwidth per LDM TCP connection? :)

daryl

--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * 
https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Fmesonet.agron.iastate.edu%2F&amp;data=04%7C01%7Caap1%40psu.edu%7C387bdd66bbff408d088b08d9194bb65a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C1%7C637568635240485745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=lo1J%2Fx7yuwtpYVExSsAvHRPdeI8PNyCAwwx7D1%2Baxlo%3D&amp;reserved=0
 */

________________________________________
From: conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Pete Pokrandt via 
conduit <conduit@xxxxxxxxxxxxxxxx>
Sent: Saturday, May 15, 2021 10:52 AM
To: support-conduit@xxxxxxxxxxxxxxxx; Anne Myckow - NOAA Federal
Subject: Re: [conduit] Recent CONDUIT latencies?

Anne,

Over the past few days, our latencies have been in line with what we have come 
to expect - 30-60 seconds as the bursts of model forecast hours come through.

Here's the graph of our latencies the past ~3 days. That red blip around 15/06 
looks like maybe a conduit ingest server's ldm was restarted? Other than that, 
they are pretty consistently 30-60s. This graph can be found at

https://nam10.safelinks.protection.outlook.com/?url=https%3A%2F%2Frtstats.unidata.ucar.edu%2Fcgi-bin%2Frtstats%2Fiddstats_nc%3FCONDUIT%2Bidd-agg.aos.wisc.edu&amp;data=04%7C01%7Caap1%40psu.edu%7C387bdd66bbff408d088b08d9194bb65a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C1%7C637568635240485745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=yDfpB6F4qLfd1kI1X9Nkc7j1fH9m5rKoGpbHjch4ljM%3D&amp;reserved=0


Pete

[cid:9377ebe1-5745-4dd7-bec0-98a991b681d2]




<https://nam10.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.weather.com%2Ftv%2Fshows%2Fwx-geeks%2Fvideo%2Fthe-incredible-shrinking-cold-pool&amp;data=04%7C01%7Caap1%40psu.edu%7C387bdd66bbff408d088b08d9194bb65a%7C7cf48d453ddb4389a9c1c115526eb52e%7C0%7C1%7C637568635240485745%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C2000&amp;sdata=LLfrnZ1DHyA%2B56YLJUHo3oVOosj35OHsYHXcVIk%2BO3U%3D&amp;reserved=0>-----
Pete Pokrandt - Systems Programmer
UW-Madison Dept of Atmospheric and Oceanic Sciences
608-262-3086  - poker@xxxxxxxxxxxx

________________________________
From: conduit <conduit-bounces@xxxxxxxxxxxxxxxx> on behalf of Anne Myckow - 
NOAA Federal via conduit <conduit@xxxxxxxxxxxxxxxx>
Sent: Tuesday, May 11, 2021 2:57 PM
To: support-conduit@xxxxxxxxxxxxxxxx <conduit@xxxxxxxxxxxxxxxx>
Subject: [conduit] Recent CONDUIT latencies?

CONDUIT users,

Can you let us know what your latencies have been recently? We have had 
customers complaining about transfer rates on some of our systems and I want to 
see if it might be affecting you as well.

Thanks,
Anne
--
Anne Myckow
Dataflow Team Lead
NWS/NCEP/NCO

• References:
  • [conduit] Recent CONDUIT latencies?
    • From: Anne Myckow - NOAA Federal
  • Re: [conduit] Recent CONDUIT latencies?
    • From: Pete Pokrandt
  • Re: [conduit] Recent CONDUIT latencies?
    • From: Herzmann, Daryl E [AGRON]