Community Input prior to CONDUIT Working Group Meeting-Dec 1999
> Hi Harry,
>
> As you probably know, there is a CONDUIT working group meeting (chaired by
> Cliff) at NASA next week. I would like to collect some information from a few
> of the Unidata sites who have been involved in this project, to find out what
> type of improvements or enhancements might be feasible, or IF the current FTP
> service from NCEP is sufficient for sites.
I did not know about this meeting (not surprising).
>
> Steve Chiswell has been working various technical issues with NWS/Office of
> Systems Operations to track down the latest problem on slow data output, which
> ended up being a failed router in OSO. This crippled the service for a long
> time and, surprisingly, we did not hear from users about the problem.
You did not hear from us before now because I thought the damage to the
Cray computer problems at NCEP were causing the grids to not come via IDD. I did not
know until yesterday that the 104 Eta grids were supposed to be back on
the IDD feed.
>
> Previously, the system would show strain at peak hours which was basically due
> to the T-1 limitation, but the current latencies are constantly pegged at 1
> hour, independent of how much trouble NCEP is currently having with their
> computers.
>
> Aside from router failures which can cause problems anywhere, this brings up
> an issue we have been grappling with. Prior to CONDUIT, users complained
> about how the FTP service to OSO and NCEP.
>
> --Can you provide us with some feed back in that regard?
> --Is it still painful to get data via FTP (is there a need for an IDD feed?)
> or have the previous network bottlenecks associated with downloading data from
> the current FTP servers been alleviated?
There is still a need for the IDD feed. FTP can still be slow, and when
the IDD feed was working we would get the grids earlier than we do now. I
strongly believe that the IDD feed is still necessary.
>
> Areas that the CONDUIT working group might be addressing are:
>
> -- the relative importance of getting the data as soon as available, e.g., for
> initializing realtime models
That is job one! This is the most time critical data that CONDUIT
transmits. And we would love higher resolution ETA data than the 104 grid.
> -- being more selective in data sets delivered by CONDUIT
I would agree to the extent that the grids needed for initialization
should have priority over all others, and that if needed, other products should be
delayed or dropped to allow the critical data through.
> -- is the primary need for the data a complete model run for offline research
> which might be acceptable through FTP
Not only research - we use whatever we get in real time for classes. And we
would still need a fast, reliable FTP server for the offline research.
> -- if you have other applications of the data, please include that information
>
Cliff can speak better to this item than I can.
> Your response is important, Harry, as the meeting takes place next week.
I'm responsible for an overview and look forward to hearing from you.
>
> Looking forward to seeing you at the AMS in Long Beach!
>
> Linda
>
> Linda Miller - lmiller@unidata.ucar.edu
> External Liaison, Unidata
> University Corporation for Atmospheric Research
> P.O. Box 3000
> Boulder, CO 80307-3000
> 303 497-8646 fax: 303-497-8690
> URL: http://www.unidata.ucar.edu/staff/lmiller/un.act.html
>
--
Dr. Harry Edmon E-MAIL: harry@atmos.washington.edu
(206) 543-0547 FAX: (206) 543-0308
Dept of Atmospheric Sciences
University of Washington, Box 351640, Seattle, WA 98195-1640
>Hi Pete,
>
>There is a CONDUIT working group meeting (chaired by Cliff Mass, Univ
>of WA) at NASA next week. I would like to collect some information
>from a few of the Unidata sites who have been involved in this
>project, to find out what type of improvements or enhancements might
>be feasible, or IF the current FTP service from NCEP is sufficient for
>sites.
Hi Linda,
I have had limited experience with the NCEP ftp service recently,
as most of the data I have been using for creating maps for
internal and web use have come in via the IDD, using both
the traditional HRS feed and the other array of products available
on the CONDUIT (NMC2 and SPARE) feed.
We did start pulling down some aviation model data from the NCEP
server for use in initializing the UW-NMS model to support a
middle atmosphere study currently underway in Sweden, and found
that the timeliness of the data and the connectivity between
the UW and NCEP are very good.
However, the hands-off aspect of receiving this data via the IDD
is very attractive. My time, as well as that of the faculty and
scientists here are limited, so it is nice to just have the data
appear automatically, rather than having to spend the time to
create and then watch over homegrown scripts used to retrieve the
same data from the ftp server at NCEP.
>Steve Chiswell has been working various technical issues with
>NWS/Office of Systems Operations to track down the latest problem on
>slow data output, which ended up being a failed router in OSO. This
>crippled the service for a long time and, surprisingly, we did not
>hear from users about the problem.
>
>Previously, the system would show strain at peak hours which was
>basically due to the T-1 limitation, but the current latencies are
>constantly pegged at 1 hour, independent of how much trouble NCEP is
>currently having with their computers.
>
>Aside from router failures which can cause problems anywhere, this
>brings up an issue we have been grappling with. Prior to CONDUIT,
>users complained about how the FTP service to OSO and NCEP.
>
>--Can you provide us with some feed back in that regard? --Is it still
>painful to get data via FTP (is there a need for an IDD feed?) or have
>the previous network bottlenecks associated with downloading data from
>the current NCEP and OSO FTP servers been alleviated?
As I mentioned above, our connetivity to NCEP via ftp seems pretty
good, in the limited amount that I have used it.
As I look right now, the NCEP2 feed (Part of the CONDUIT project) is
running about 25 min behind. With the amount of data coming over that
stream (120 Mb/hour or so) I am not surprised to see it fall somewhat
behind, but as long as it remains above the 1 hour threshold where we
begin to lose data that has been deleted from upstream queues, this type
of delay wouldn't really bother me. It would probably take a similar
amount of time to retrieve the data via FTP.
I'll try to keep a closer watch on our lag for the CONDUIT feeds
and offer feedback on how it is doing.
>
>Areas that the CONDUIT working group might be addressing are:
>
>-- the relative importance of getting the data as soon as available,
> e.g., for initializing realtime models
One problem with using IDD delivered data to initialize models is that
because the individual grib records are sent one by one, it is very
common to have a data set that is missing one or two grids, for example
the 600 hPa U component of the wind may be missing, while all the other
grids are there. It is possible to get the model initialization to
handle missing data such as this, but it's much easier to just grab a
complete file from NCEP, so you know that it is complete, and initialize
models off of that.
I usually tell my users to use the IDD data for plotting and analysis
where missing data is usually not as big of an issue, but to get a
complete data set from somewhere (NCEP, NCAR or one of our local
archives) via ftp to initalize models with.
>-- being more selective in data sets delivered by CONDUIT
>-- is the primary need for the data a complete model run for
offline research which might be acceptable through FTP
Because of the potential of missing data in IDD datasets, anyone
here who is doing model runs has made arrangements on their own
to retrieve the specific data set that they use to initialize with.
However, the maps that we put on our web site, as well as the paper
maps that we are locally plotting, to replace the DIFAX maps (which we
stopped receiving in January, 1999) are created exclusively from data
ingested via the IDD.
One data set that we had been getting until roughly the time of the NCEP
computer fire was output from several Ensemble forecast runs. We had a
nice product going on the web and on paper, which was a 4 panel display
of MRF fields, as well as a spaghetti plot for specified 500 hPa height
contours, to give an overview of the main MRF run and how it compares to
the other members of the ensemble. This way one can have some degree of
confidence on what the MRF is portraying.
This dataset was coming in on the SPARE feed (the one Steve
Chiswell was maintaining I believe). Our pqact entry looks like:
SPARE ^ens.*/F.../HGT/.*
PIPE /home/gempak/NAWIPS-5.4/bin/linux/dcgrib
-d /usr/local/ldm/logs/dcgrib_oso.log
-t 60 -g /home/gempak/NAWIPS-5.4/gempak5.4/tables -m 9999
PACK /u3/ldm/data/gemdata/hds/YYMMDDHH_ens.gem
I'll contact Steve personally about the status of the Ensemble
data.
>-- if you have other applications of the data, please include
> that information
Mainly just the creation of hard copy maps to replace the DIFAX
map feed. Michael Morgan (a faculty member here, who is on the
Unidata Users committee) had spoken to me about the possibility
of broadcasting our maps in (perhaps gzip'd) postscript format
via the IDD to other sites that are concerned about the loss
of DIFAX maps at some time in the future. I'd be happy to work
with someone at Unidata to set up such a feed.
We are currently producing North American region analyses for surface
and mandatory upper air pressure levels, max/min temperatures, 24 hour
precip, snow cover, and a composite moisture plot; Northern and Southern
Hemispheric thickness/MSLP maps; and various multi-panel maps for the
Eta, NGM, Aviation, MRF, and (when it is coming in) Ensemble model runs,
including NGM, AVN and MRF MOS output.
>
>Your response is important, Pete, as the meeting takes place next
>week. I'm responsible for an overview and look forward to hearing from
>you.
I hope this is helpful. Please feel free to contact me should
you wish to discuss any of these issues further.
Pete
--
+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+
^ Pete Pokrandt V 1447 AOSS Bldg 1225 W Dayton St^
^ Systems Programmer V Madison, WI 53706 ^
^ V poker@meteor.wisc.edu ^
^ Dept of Atmos & Oceanic Sciences V (608) 262-3086 (W) 222-0919 (H) ^
^ University of Wisconsin-Madison V 262-0166 (Fax)262-3086 (VM)^
<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<+>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>+
Hi, Cliff:
We have a lot of interest in the CONDUIT data stream and have been a test
site since its inception almost two years ago. Unfortunately, we have not
had much luck in successfully receiving the products in the last 6-9
months. Between the high latencies exhibited almost every single day
(latencies over an hour), the current LDM limitations, the router problems
at OSO, and the NCEP fire, the reception has been so poor that we had lost
interest in using the CONDUIT data. I would very much like to see a
RELIABLE CONDUIT stream that can be used for a variety of research and
educational purposes. However, it has been extremely frustrating because
of all the problems. If we (Unidata, NCEP, USWRP, and universities) can't
solve this problem, how are we going to cope with providing other large
volume data sets (higher resolution models, WSR 88-D, etc.) to the
academic community in the future?
We do use the NCEP FTP servers every single day and the performance has
been satisfactory. We will continue to use that as our main mode of
gridded data access until the CONDUIT problems are resolved. The products
we are most interested are the high resolution Eta and RUC-II data, along
with the MRF ensembles when NCEP is running them again. We also FTP the
AVN data from NCEP for initializing our realtime regional spectral model
runs.
I would make a strong recommendation fund this effort (not just lip
service) and ensure that these problems are resolved. At the moment, this
effort is done in a ad hoc manner on people's own time. As a result, this
has not been a high priority issue for many of the principals. And the
result shows...
I have also expressed these concerns and my frustrations during a phone
conversation with Linda Miller earlier today.
Anyway, that is my 2 cents.
Cordially,
Mohan
At 05:53 PM 12/2/99 -0800, you wrote:
>Dear Colleagues,
> A major issue that was discussed during the last Real-Time Workshop was
> the difficulty in getting NCEP grids reliably from NCEP's ftp
> servers. In response, another mode of delivery was established under the
> UNIDATA/USWRP/NASA/ NOAA mantel... internet delivery (IDD) using the
> UNIDATA LDM system. In a week a meeting in DC will reconsider this
> issue. Thus, my question to those of you making use of real-time NCEP
> grids for local NWP and other uses:
>
>1. Is the Conduit/IDD system working for you?
>2. Is the performance of the NCEP ftp servers acceptable to you now? Are
>there still problems with ftp access?
>3. What NCEP products that you are not getting now would you like?
>
>Any feedback on the above or related topics is very welcome.
>
>Thanks, Cliff Mass
>
>
>
>
>Professor Clifford F. Mass
>Department of Atmospheric Sciences, Box 351640
>University of Washington
>Seattle, Washington 98195
>
>email: cliff@atmos.washington.edu
>Telephone: (206) 685-0910 (Voice), (206) 543-0308 (Fax)
----------------------------------------------------------------------------
Dr. Mohan Ramamurthy Email:
mohan@uiuc.edu
Associate Professor Ph: (217)
333-8650
Department of Atmospheric Sciences Fax: (217)
244-4393
University of Illinois at Urbana-Champaign
105 S. Gregory Street
Urbana, IL 61801-3070
Linda,
I'm way behind on reading my "bulk" email and embarrassed that I
inadvertently overlooked the included note below. Though I'm too late for
the referenced meeting, I will still make a few comments...
On Wed, 1 Dec 1999, Linda Miller wrote:
> There is a CONDUIT working group meeting (chaired by Cliff Mass, Univ of WA) at
> NASA next week. I would like to collect some information from a few
> of the Unidata sites who have been involved in this project, to find out what
> type of improvements or enhancements might be feasible, or IF the current FTP
> service from NCEP is sufficient for sites.
I know a lot of folks here use ftp from NCEP *because* they found it was
faster than what we had available from Unidata.
> Steve Chiswell has been working various technical issues with NWS/Office of
> Systems Operations to track down the latest problem on slow data output, which
> ended up being a failed router in OSO. This crippled the service for a long
> time and, surprisingly, we did not hear from users about the problem.
I would complain from time-to-time, but with the many other things I have
to attend to, I don't always notice right away when data is not coming in
as it should. At some point, the CONDUIT data became so variable that I
felt complaining would not help because it was a known problem that
Unidata was working on and I don't like to be obnoxious in such cases. I
did get some pressure from one group in our department disatisfied with
the delivery, but they discovered that ftp worked and I think have pretty
much jumped ship on using the Unidata CONDUIT stuff.
> Previously, the system would show strain at peak hours which was basically due
> to the T-1 limitation, but the current latencies are constantly pegged at 1
> hour, independent of how much trouble NCEP is currently having with
>their computers.
>
> Aside from router failures which can cause problems anywhere, this brings up
> an issue we have been grappling with. Prior to CONDUIT, users complained about
> how the FTP service to OSO and NCEP.
>
> --Can you provide us with some feed back in that regard?
> --Is it still painful to get data via FTP (is there a need for an IDD feed?)
> or have the previous network bottlenecks associated with downloading data from
> the current NCEP and OSO FTP servers been alleviated?
I think you've already discussed this at the meeting, but I think there
*is* a need for one (or more) IDD feed(s), unless it's decided that push
technology is no longer important (hah). I think it's a tremendous
benefit to have data families delivered immediately when ready. For each
site (and possibly multiple users within each site) to have to set up a
data pulling system is a waste of time and effort in most instances.
However, I think our researchers will have to feel comfortable about the
reliability of the data delivery before they will switch back. Therefore,
for CONDUIT to work, I think there will have to be a high committment to
the reliable delivery of the data by all parties involved.
> Areas that the CONDUIT working group might be addressing are:
>
> -- the relative importance of getting the data as soon as available, e.g., for
> initializing realtime models
> -- being more selective in data sets delivered by CONDUIT
> -- is the primary need for the data a complete model run for offline research
> which might be acceptable through FTP
> -- if you have other applications of the data, please include that information
>
> Your response is important, Art, as the meeting takes place next week.
>I'm responsible for an overview and look forward to hearing from you.
I apologize for missing this important email when it arrived. If you need
my input again and don't get a quick response, don't hesitate to send a
reminder.
Art.
Arthur A. Person
Research Assistant, System Administrator
Penn State
