[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20050131: [Fwd: Re: Late 1 degree GFS and late eta's on CONDUIT] (fwd)

>From:  Tiziana Cherubini <address@hidden>
>Organization:  University of Hawaii
>Keywords:  200502010005.j1105Fv2005162 IDD CONDUIT

Hi Tiziana,

>       how are you?

I am well, thank you.  I hope you are well also.

>I'm still in Hawaii and actually just came
>back from what was supposed to be a vacation: 2 weeks back home
>(Italy) over Christmas which becames a 5 weeks stay due to visa
>problems. I had to be double checked by the US security program
>because my name is the same that a ~50 years old italian red
>brigade terrorist who's currently in jail :) Isn't this a funny
>story?

Well, it is not really funny; it is kind of sad actually.  I hope that
your 5 weeks home was enjoyable :-).

>       Anyway, I'm writing you because of some problems we are
>experimenting with the LDM. I know you might not be the one to
>ask but I hoped you could explain me better what's going on or
>point me the right person to talk to.

OK.

>It looks like during January, 50% of the times, the 00Z and 12Z
>GFS data disseminated through LDM came in late (about 2hours
>later than the normal time). Late enough that we are having
>troubles starting timely our MM5 forecasts
>(we use GFS as initial conditions). As from the email below
>it looks like there have been changes to the dissemination
>procedures and they are working on it to fix the delay. I
>was just wondering whether people there at unidata have an
>idea on how long would it take before things get back to
>normal.

The major problem is that the machine that is doing the injection of
the CONDUIT data products is just too slow to handle the volume of data
that is now included in the stream.  What happens is that the data is
waiting a long to be injected into the LDM queue so users are not
getting the products as quickly as they would like/need.

As Steve Chiswell (Chiz) pointed out in the email he sent to the
CONDUIT users email list (and which you attached below), we tried a
different approach to injecting the data and were successful in
dramatically cutting down the time that it took for end users to
receive that data, but we received complaints from users whose use of
the data was based on the grib messages arriving in a fixed order.
Because of the complaints, we had to back of off the parallel insertion
test.  Chiz told me that someone at NCEP is working on a scheme to
serialize the data flow when using parallel queue insertion.  We will
have to wait to see if his effort will work or not.  The only other
thing that would help the effort out is for NCEP to install a faster
injection machine, but that is not likely to happen given the extremely
tight budget situation.

>We also tried using the NCEP ftp but it's way too slow...
>perhaps too many user trying to achieve data timely.

Exactly.  The NWS/NCEP is interested in seeing the LDM distribution
of the CONDUIT data work since it might help alleviate the bottleneck
that is created by lots of users FTPing the data at basically the
same time.

>Thank you very much for any information you could provide us.

I can assure you that we are thinking about how we can make the delivery
of the CONDUIT data faster, and will test/implement anything that makes
sense.  We have no control, however, on the speed of the injection
machine (sigh).

>Aloha,
>               Tiziana
>
>---------------------------------------------------------------
> Tiziana Cherubini, Ph.D.             Department of Meteorology
> Research Meteorologist               University of Hawaii
> Mauna Kea Weather Center             2525 Correa Rd. HIG 367
> http://mkwc.ifa.hawaii.edu           Honolulu, Hawaii 96822
>
> tel.: (808) 956-4593                 fax : (808) 956-2877
>---------------------------------------------------------------

Ciao,

Tom

>---------- Forwarded message ----------
>Date: Mon, 31 Jan 2005 10:24:22 -1000
>From: Ryan Lyman <address@hidden>
>To: Tiziana Cherubini <address@hidden>
>Subject: [Fwd: Re: Late  1 degree GFS and late eta's on CONDUIT]
>
>
>-- 
>
>-------------------------------------------------------
>  Ryan Lyman                   Mauna Kea Weather Center
>  Institute for Astronomy      Forecast Meteorologist
>  640 North A'ohoku Place      Tel: (808) 932-2323
>  Hilo, Hawaii 96720-2700      Fax: (808) 933-0737
>
>              http://mkwc.ifa.hawaii.edu/
>-------------------------------------------------------
>
>
>--Boundary_(ID_E+c2nkI8nc9HRO4AKV8U8A)
>Content-id: <Pine.GSO.4.33.0501311147083.13359@tutt19>
>Content-type: message/rfc822;
> NAME="Re: Late 1 degree GFS and late eta's on CONDUIT"
>Content-description: 
>
>Return-path: <address@hidden>
>Received: from conversion-daemon.mail.hawaii.edu by mail.hawaii.edu
> (iPlanet Messaging Server 5.1 HotFix 1.14 (built Oct  8 2003))
> id <address@hidden> for rlyman@ims-ms-daemon
> (ORCPT address@hidden); Fri, 14 Jan 2005 12:36:29 -1000 (HST)
>Received: from mta01.hawaii.edu (mta01.hawaii.edu [128.171.94.89])
> by mail.hawaii.edu
> (iPlanet Messaging Server 5.1 HotFix 1.14 (built Oct  8 2003))
> with ESMTP id <address@hidden> for rlyman@ims-ms-daemon
> (ORCPT address@hidden); Fri, 14 Jan 2005 12:36:14 -1000 (HST)
>Received: from unidata.ucar.edu (laraine.unidata.ucar.edu [128.117.140.62])
>       by mta01.hawaii.edu (8.11.7p1+Sun/8.11.7) with ESMTP id j0EMaB026160; F
> ri,
> 14 Jan 2005 12:36:11 -1000 (HST)
>Received: (from majordo@localhost)     by unidata.ucar.edu (UCAR/Unidata)
> id j0EMYx4L005076     for conduit-out; Fri, 14 Jan 2005 15:34:59 -0700 (MST)
>Received: from flip.unidata.ucar.edu (flip.unidata.ucar.edu [128.117.140.85])
>       by unidata.ucar.edu (UCAR/Unidata) with ESMTP id j0EMYqv2005040; Fri,
> 14 Jan 2005 15:34:52 -0700 (MST)
>Received: (from chiz@localhost)        by flip.unidata.ucar.edu (UCAR/8.12.5/S
> ubmit)
> id j0EMYqWe6972642; Fri, 14 Jan 2005 15:34:52 -0700 (MST)
>Date: Fri, 14 Jan 2005 15:34:52 -0700
>From: Steve Chiswell <address@hidden>
>Subject: RE: Late  1 degree GFS and late eta's on CONDUIT
>In-reply-to: <address@hidden>
>Sender: address@hidden
>To: address@hidden
>Cc: address@hidden
>Reply-to: address@hidden
>Message-id: <address@hidden>
>Organization: Unidata
>MIME-version: 1.0
>X-Mailer: Ximian Evolution 1.2.0
>Content-type: text/plain; charset=ISO8859-1
>Content-transfer-encoding: 7bit
>Precedence: bulk
>X-MailScanner: Found to be clean
>X-MailScanner-SpamCheck: not spam, SpamAssassin (score=0.282, required 5,
>       ORDER_NOW)
>References: <address@hidden>
>Keywords: 200501142234.j0EMYqv2005040
>X-Authentication-warning: flip.unidata.ucar.edu: chiz set sender to
> address@hidden using -f
>
>
>CONDUIT data users, 
>We are actively working on a solution to increase the throughput of data
>products and decrease the queueing time of these products from the NWS
>file server to the LDM queue.
>
>On January 5th, the product insertion for the CONDUIT data stream was
>changed to allow product insertion in parallel. This method increased
>the aggregate throughput from a maximum of 1.8GB per hour, to
>2.5 to 2.7 GB per hour at the 4 peak times. However, a number of users
>stated that this method was troublesome for their processing of forecast
>hour products since while the aggregate data volume was increased, the
>arrival time of specific forecast times was intermixed and the earliest
>forecast times were not guaranteed to complete prior to other times.
>
>Several alternatives were tried to maintain product order while
>preserving increased throughput, but products were lost in queueing
>order, so this process was halted as soon as these consequences were
>observed. On Thursday morning, the CONDUIT insertion queue was restored
>to the synchronus remote method that existed at the beginning of
>December when the 0.5 degree GFS files were added. This return was 
>seemingly the most preferable prior to a holiday weekend, from input to
>the conduit mail list so as to not have "missing products" while an
>improvement in timeliness was being sought. The queueing backlog for
>this was known to exceed an hour and a half.
>
>The largest backlog of queuing occurs when the twice daily ensembles are
>posting at the time that the 4x per day GFS and ETA are posting, as
>well as the hourly RUC. This begins be delaying the posting the 12Z GFS
>and 18Z NAM and continues through the peak posting when grids are
>posting to the file server at the rate several models and multiple grids
>per minute.
>
>Early next week a change will be made to isolate the underlying file
>system from product insertion which will hopefully improve timeliness.
>
>Steve Chiswell
>Unidata User Support
>
>
>
>
>
>
>On Fri, 2005-01-14 at 14:47, Robert Mullenax wrote:
>>  It would be nice to go back to pre-addition of 0.5 deg GFS.
>> 
>> -----Original Message-----
>> From: address@hidden
>> To: Jerrold Robaidek
>> Cc: Conduit Users; address@hidden
>> Sent: 1/14/2005 1:16 PM
>> Subject: Re: Late  1 degree GFS and late eta's on CONDUIT
>> 
>> Oh yes, we are seeing huge delays.  As an example, we received last
>> night's 0Z run 72 hour forecast for the GFS at 2 hours and 54 later
>> than I was able to obtain the file via FTP (23:01:58 PST vs 20:08:15
>> PST). 
>> 
>> 
>> On Fri, Jan 14, 2005 at 11:56:50AM -0600, Jerrold Robaidek wrote:
>> > 
>> > 
>> > The 12 Z GFS 1 degree just started coming in and the NAMs (etas) are
>> very 
>> > late....
>> > 
>> > Things seem to have gone from bad to worse ....
>> > 
>> > (I do have some latency issues, but only 10 minutes, not the hours
>> late 
>> > that we are seeing.)
>> > 
>> > Anybody else seeing these problems?
>> > 
>> > 
>> > Jerry
>> > 
>> > -- 
>> > Jerrold Robaidek                       Email:  address@hidden
>> > SSEC Data Center                       Phone: (608) 262-6025
>> > University of Wisconsin                Fax: (608) 263-6738
>> > Madison, Wisconsin
>
>--Boundary_(ID_E+c2nkI8nc9HRO4AKV8U8A)
>Content-id: <Pine.GSO.4.33.0501311147084.13359@tutt19>
>Content-type: message/rfc822; NAME="CONDUIT and data volume implications"
>Content-description: 
>
>Return-path: <address@hidden>
>Received: from conversion-daemon.mail.hawaii.edu by mail.hawaii.edu
> (iPlanet Messaging Server 5.1 HotFix 1.14 (built Oct  8 2003))
> id <address@hidden> for rlyman@ims-ms-daemon
> (ORCPT address@hidden); Wed, 26 Jan 2005 08:33:28 -1000 (HST)
>Received: from mta01.hawaii.edu (mta01.hawaii.edu [128.171.94.89])
> by mail.hawaii.edu
> (iPlanet Messaging Server 5.1 HotFix 1.14 (built Oct  8 2003))
> with ESMTP id <address@hidden> for rlyman@ims-ms-daemon
> (ORCPT address@hidden); Wed, 26 Jan 2005 08:32:44 -1000 (HST)
>Received: from unidata.ucar.edu (laraine.unidata.ucar.edu [128.117.140.62])
>       by mta01.hawaii.edu (8.11.7p1+Sun/8.11.7) with ESMTP id j0QIWep24608; W
> ed,
> 26 Jan 2005 08:32:40 -1000 (HST)
>Received: (from majordo@localhost)     by unidata.ucar.edu (UCAR/Unidata)
> id j0QIVfxU026548     for conduit-out; Wed, 26 Jan 2005 11:31:41 -0700 (MST)
>Received: from flip.unidata.ucar.edu (flip.unidata.ucar.edu [128.117.140.85])
>       by unidata.ucar.edu (UCAR/Unidata) with ESMTP id j0QIVdv2026544; Wed,
> 26 Jan 2005 11:31:39 -0700 (MST)
>Received: (from chiz@localhost)        by flip.unidata.ucar.edu (UCAR/8.12.5/S
> ubmit)
> id j0QIVc9N1131564; Wed, 26 Jan 2005 11:31:38 -0700 (MST)
>Date: Wed, 26 Jan 2005 11:31:38 -0700
>From: Steve Chiswell <address@hidden>
>Subject: CONDUIT and data volume implications
>In-reply-to: <address@hidden>
>Sender: address@hidden
>To: David Knight <address@hidden>
>Cc: address@hidden, address@hidden
>Reply-to: address@hidden
>Message-id: <address@hidden>
>Organization: Unidata
>MIME-version: 1.0
>X-Mailer: Ximian Evolution 1.2.0
>Content-type: text/plain; charset=ISO8859-1
>Content-transfer-encoding: 7bit
>Precedence: bulk
>X-MailScanner: Found to be clean
>X-MailScanner-SpamCheck: not spam, SpamAssassin (score=0, required 5)
>References: <address@hidden>
>Keywords: 200501261831.j0QIVdv2026544
>X-Authentication-warning: flip.unidata.ucar.edu: chiz set sender to
> address@hidden using -f
>
>David,
>
>Regarding downstream bottlenecks on increased data volumes:
>
>One item that is of particular importance to IDD sites is the use of
>well formed regular expressions in ldmd.conf request lines.
>All sites feeding downstream IDD sites should periodically review the
>connection patterns in their ldmd.log files for the request patterns
>their downstream are making (as well as checking their own ldmd.conf
>request lines when changes are made). The LDM distribution contains a
>test program called "regex" and provides a man page on usage and
>details on performance as well. For instance, if you are using "split"
>feed request pattens for CONDUIT, the two versions of request patterns
>
>.*[09]$
>or
>[09]$
>
>have vastly different performance impacts on the upstream ldm which is
>trying to determine if a product matches the pattern you are requesting.
>I testing this pattern using regex with 10,000 iterations,
>the .*[09]$ pattern used 49 cpu seconds, while [09]$ used <1 cpu
>seconds. The point to remember here is that it is your downstream that
>can impact your performance using bad regex patterns.
>
>I will provide a synopsis of CONDUIT data stream content possibilities
>for consideration under separate email.
>
>Steve Chiswell
>Unidata User Support
>
>
>
>
>
>There are several items for all LDM sites to be aware of that affect 
>LDM performance
>
>
>
>On Wed, 2005-01-26 at 08:41, David Knight wrote:
>> Hi all,
>> > 
>> > NCEP and UNIDATA continue to work the issue of the delays on CONDUIT.  
>> > We hope to have a solution ready for implementation in 2-3 weeks.  
>> > Please hang in there.
>> > 
>> I'm wondering what the plan is, and what the possible
>> side effects are.
>> Specifically I'm concerned that if NCEP can really start
>> cramming all this stuff through their LDM in a timely
>> fashion that the bottleneck will just move downstream
>> to the receiving sites that may not be able to deal
>> with this volume of data all at once.
>> One possible alternative would be to delay insertion
>> of the the 0.5 deg GFS until the NCEP queue has
>> been mostly processed. I'm not sure if this is
>> practical, or desirable, but there maybe some
>> benefit in spreading out the feed over time rather
>> than have it be so eposodic.
>> Just throwing this out as a possible iten to consider
>> and discus.
>> David
>> 
>> David Knight
>> Department of Earth and Atmospheric Sciences   Tel: (518)-442-4204
>> University at Albany   ES-228                  Fax: (518)-442-4494
>> Albany, NY  12222                              Email: address@hidden
> u
>
>--Boundary_(ID_E+c2nkI8nc9HRO4AKV8U8A)--
--
+-----------------------------------------------------------------------------+
* Tom Yoksas                                             UCAR Unidata Program *
* (303) 497-8642 (last resort)                                  P.O. Box 3000 *
* address@hidden                                   Boulder, CO 80307 *
* Unidata WWW Service                             http://www.unidata.ucar.edu/*
+-----------------------------------------------------------------------------+