[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: CONDUIT feed timing

In a previous message to me, you wrote: 

 >
 >Jerry,
 >
 >Yesterday at 1830Z we implemented a parallel queueing scheme at the
 >NWS that we hope will improve the timeliness of data being ingected into
 >the CONDUIT data stream. Any feedback you can provide on how
 >this affects your reception would be greatly appreciated.
 >
 >Since data will be inserted in parallel, you will notice that multiple 
 >model runds and forecast times will probably be interspersed where
 >previously they had been serialized.
 >
 >I watched the 00Z GFS last night, and the posting gap between f084 and
 >f132 was matched on the FTP server at 0422Z and later at 0509Z, the
 >other grids were posted to the NWS servers, so all appears to be
 >behaving correctly on this end.
 >
 >Steve Chiswell
 >Unidata User Support

Steve,

Since the parallel queueing scheme was implemented, I have noticed two
things, both are problematic.

1) The lag goes way up since so much more data is being inserted into
the queue in a shorter amount of time. I was getting lag times of 3000-4000
seconds in peak gfs times.  I moved my ldm queue on f5 to a faster disk, and
that helped, but it's still getting up to 300-400 seconds.

2) The biggest problem I see, is that now it takes MUCH longer for the
early hours of a forecast run to complete.  Most of my model plotting
scripts and all of our real-time mesoscale modelling efforts here
take advantage of the fact that an entire model run doesn't need to be
complete before we can start.  Since the parallel queueing was enabled,
the 00 hour forecast of the eta model for example, takes over an hour
to get here, where previously it was taking maybe 5 minutes.

If this is how it's going to be, I'd much prefer the old serial ingest,
or maybe some kind of blend, so the first 12 hours or so (or 48 or 
whatever) of a model run can get priority. 

It really hurts us to have the 00 hour forecast complete around the 
same time as the later forecasts, even if the entire model run gets 
in faster as a result.

For what it's worth.

Pete


--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt                    V 1447  AOSS Bldg  1225 W Dayton St^
^ Systems Programmer               V Madison,         WI     53706    ^
^                                  V      address@hidden       ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (Phone/voicemail) ^
^ University of Wisconsin-Madison  V       262-0166 (Fax)             ^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+