Hi Paul (and Brian, Tim, and Vic), Several of us have just finished discussing your inquiry, and this reply is what we all agree on: Paul asked: > A side question. Considering the behavior I'm seeing, do you think > changing the feedset for the grb2 data might help? No. The feedtype used should have no effect on the problems you are experiencing. Paul asked: > For our setup we do not see any data on the NMC feedset and have > considered using it as the sole source for the grb2 data. Thoughts? If your setup does not interact with the Unidata IDD, then you are free to use any feedtype you find convenient. However, if there is any possibility of your LDM interacting directly or indirectly with the IDD, then we ask that you do not use a feedtype whose content is already defined by existing use. The feedtypes open for use are currently EXP and SPARE. NB: the NMC feedtype is a "compound" feedtype: it is the union of the AFOS, NMC2, and NMC3 (raw feedtypes FT11, FT12 and FT13). It is best to use a single feedtype when creating your own datastream. Steve asked if there is more than one REQUEST entry in the LDM configuration-file on Ldad. The ldmd_ldad.conf file you sent as an attachment to one of your emails answered this question: - you have multiple ldmd.conf REQUEST lines for different kinds of data you are putting in the EXP feedtype - all of the GFS 0.5 degree data (.grb2) is being requested on a single ldmd.conf REQUEST line Comment: It is our belief that the firewall between your ldad and arps LDM is causing the slowdown in receipt of GRIB2 products in your EXP data feed. The firewall could be throttling the connection based on the high volume of data in that connection, or the firewall could be performing "deep inspection" of each product (each packet of each product) being sent. In either case, the serialization of the data products would lead to monotonically increasing latencies. From our perspective, there are two ways of addressing this: - change the firewall configuration to stop the slowdown/deep inspection of the products - split your ldmd.conf REQUEST for .grb2 products into several REQUESTs Comments: - the first option may not be possible/allowed due to security policies at JSC. I would, nonetheless, pursue this with all possible vigor as it is the most straightforward way of solving the problem. Questions: - are both ldad and arps machines internal to JSC? - if yes, is there really a need to throttle the connection or deeply inspect each packet? - splitting the single REQUEST for .grb2 data would necessitate there being something in the product IDs that would allow the set of products to be separated into 'n' mutually exclusive subsets. We do this with the IDD CONDUIT feed by including a sequence number as a field in the product ID. For example, here are the product IDs for three successive CONDUIT products: data/nccf/com/nam/prod/nam.20091006/nam.t18z.grbgrd03.tm00.grib2 !grib2/ncep/NAM_84/#000/200910061800F003/AVOR/500 Pa PRES! 000003 data/nccf/com/nam/prod/nam.20091006/nam.t18z.grbgrd03.tm00.grib2 !grib2/ncep/NAM_84/#000/200910061800F003/AVOR/1000 Pa PRES! 000004 data/nccf/com/nam/prod/nam.20091006/nam.t18z.grbgrd03.tm00.grib2 !grib2/ncep/NAM_84/#000/200910061800F003/CINS/0 - NONE! 000005 The final field is a 6 sequence number that we use to subset the products into mutually exclusive subsets. Since you are generating your own product IDs, it should be easy to add a sequence number that allows for the same kind of sub-setting. I suggest that you first try adding sequence a number that ranges from 000 to 009, and the restarts (i.e., 000, 001, ..., 009, 000, ...). A 5-way split of your current single REQUEST could look like: change: request EXP ".*.grb2" 188.8.131.52 <- NB: the leading .* is not needed to: request EXP "\.grb2.*$" 184.108.40.206 request EXP "\.grb2.*$" 220.127.116.11 request EXP "\.grb2.*$" 18.104.22.168 request EXP "\.grb2.*$" 22.214.171.124 request EXP "\.grb2.*$" 126.96.36.199 The idea behind splitting the single REQUEST into multiple REQUESTs is to drop the volume of each REQUEST stream which may be enough to fall "under the radar" of the firewall as it is currently setup (if volume is triggering a throttling by the firewall), or to have the firewall work on more, lower volume streams simultaneously (if the firewall is doing deep inspection). If a 5-way split does not drop the latencies enough, you could do a 10-way split: request EXP "\.grb2.*0$" 188.8.131.52 request EXP "\.grb2.*1$" 184.108.40.206 request EXP "\.grb2.*2$" 220.127.116.11 request EXP "\.grb2.*3$" 18.104.22.168 request EXP "\.grb2.*4$" 22.214.171.124 request EXP "\.grb2.*5$" 126.96.36.199 request EXP "\.grb2.*6$" 188.8.131.52 request EXP "\.grb2.*7$" 184.108.40.206 request EXP "\.grb2.*8$" 220.127.116.11 request EXP "\.grb2.*9$" 18.104.22.168 If splitting the feed helps but a 10-way split still does not reduce latencies to acceptable levels, you will need to create a larger sequence number (e.g., 000 to 099) and split the feed further. Paul wrote: > while watching this yesterday I did an "ls -l" on the ldm.pq > file and it indicated it was ~2.6GB. This was the indicated > size prior to the 12Z model run. After watching the 12Z data > I looked at the file again and it ndicated it was ~3.9GB. The > default queue size defined in ldmadmin-pl.conf at the time was > set to "1G". Comments: - we are totally at a loss to explain this behavior as it: - is not how things should work - has _never_ been reported by any site - a LONG time ago (early LDM-5 days), the LDM queue could grow, but that was only on a machine running IRIX. We have run the LDM-6 on IRIX for a _long_ time (several years) and have never seen it change size Questions: - what OS is running on your ldad machine (providing the output of 'uname -a' would be very helpful)? - is your LDM queue on an NFS-mounted, RAID or local file system? We strongly recommend that the LDM queue be put on a local file system. Under Solaris, putting the queue on a RAID seems to work fine, but we have found this to be problematic under Linux (I tested putting the LDM queue on both a software and hardware RAID under Fedora Linux a few years back. My tests showed that product latencies would rapidly grow to the maximum latency allowed (default is 3600 seconds) even when feeding from internal machines in the UPC. When I moved the queue off of the RAID and onto a local file system (ext2 or ext3), the latencies would drop to less than one second). Paul wrote: > We recently upgraded our AWIPS to OB9.1 and since that time we > have been having problems getting our GFS 0.5 degree grid data > from our MIDDS/McIDAS ldm Comment: - our guess is that the configuration of the firewall between your ldad and arps machines was changed at or about the same time that your AWIPS installation was upgraded Paul wrote: > As for particulars we are running ldm 6.6.5 on both sides. Comment: - the current LDM release is 6.8.1, but we do _not_ believe that this has anything to do with the problems you are experiencing Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: NGQ-500529 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.