Growing Demands on LDM resources
Steve Chiswell
May 11, 1997
This winter, Unidata began the process of
increasing the resolution of satellite data
available on the IDD as requested by the User Committee. By increasing the resolution
of the imagery from 8km to 4km, the resultant size of the images transmitted
is increased by a factor of four, therefore it was decided to gradually phase in
the increase of resolution so that sites may assess the impact of this change.
In announcing the pending change, Tom Yoksas asked for feedback from sites as
to how this change would effect them.
In response to the first of the series of resolution increases, several sites
responded that the higher resolution images were detrimental to their
LDM performance and as a result, many of the images were not being received.
As a result of this feedback, the User Committee discussed the pros and
cons of increased resolution at the 13-14 March, 1997 meeting and asked
that Unidata evaluate the transmission of these images.
In order to better view the state of site IDD reception, I developed a
GEMPAK program to plot the latency statistics being produced by Robb Kambic.
Following the User Committee meeting, Jim Koermer at Plymouth State
again expressed his displeasure with his reception of the larger
images and requested that other images not be increased in resolution.
I reviewed the IDD latencies of data being received at Plymouth
State and found that in fact, data reception at Plymouth was
very timely, and in this case it was not a problem with
networking as had been presumed. After tracking several products
in transmission from the upstream site at UIUC to Plymouth using the
LDM notifyme command, and watching the ldmd.log file
being posted on PSC's web pages, I found that the products were
actually reaching Plymouth, but being dropped locally.
Apr 17 15:43:04 mammatus data[19000]: Deleting oldest to make space 1926376 bytes
Apr 17 15:43:05 mammatus data[19000]: del_oldest: conflict on 23209544
Apr 17 15:43:05 mammatus data[19000]: comings: pqe_new: Permission denied
Apr 17 15:43:05 mammatus data[19000]: : 2f4c12cd54cf92502e2299d4f3cce7dd
1926268 19970417154214.529 MCIDAS 000 LWTOA3 143 DIALPROD=UV 97107 153850
Apr 17 15:43:06 mammatus data[19000]: Connection reset by peer
Apr 17 15:43:06 mammatus data[19000]: Disconnect
As the above log information shows, the Plymouth LDM was suffering from
a lack of available space in their LDM product queue, and the
larger products were being dropped. In actuallity, PSC
receives quite a bit of data, including WSI NIDS. As satellite
imagery resolution increases, the demand on the amount of
space required by the local product queue also increases. When
a product queue is too small, a taxing burden is placed on the LDM which
must scour space in order to store a product.
When new data is added to the IDD, sites must be made aware that they
should review their product queue usage and adjust the size accordingly.
In this case, detection of the problem was greatly simplified by
the site providing their LDM log files on the web, and by
sending their stats files in to the UPC. I encourage
all sites that do not currently mail their stats to the UPC and
post their LDM logs to please endeavor to do so. If you do not
see your latency stats plotted on our web pages, please consult
with support staff at Unidata to track down where the data is getting lost.
