>From: David YEUNG <address@hidden> >Organization: CCAR HKUST >Keywords: 200211142212.gAEMCPL03300 IDD LDM Hi David, >This is the information regarding our LDM: > fully qualified hostname > >rcz006.ust.hk > > LDM version > >ldm-5.1.4 > > fully qualified name of the upstream host > >iita.rap.ucar.edu > > datastreams you are trying to feed > >WMO > > > o copies of your ldmd.conf and pqact.conf files > >See the attached files. > > > o what operating system (vendor, release, etc.) you are running on > > your LDM machine > >RedHat 7.2 > >address@hidden ldm]$ uname -a >Linux rcz006.ust.hk 2.4.9-13 #1 Tue Oct 30 20:11:04 EST 2001 i686 unknown Thanks for this information. I see right off a problem you will see with the current incarnation of the LDM. Your feed request line: # LDM5 servers request data # # request <feedset> <pattern> <hostname pattern> # request WMO ".*" iita.rap.ucar.edu will cause all of the products in the 'WMO' feed to be delivered on one rpc.ldmd connection. This works fine for machines that are electronically close to their upstream feed site(s), but this is not the case for you. A little bit of background by way of explanation is in order: the current LDM (including versions up to and including 5.2.2) uses a blocking RPC to deliver data products that are less than 16384 bytes in size and multiple blocking RPC calls for products that are larger than 16384 bytes. A blocking RPC call is one where the upstream LDM waits until it receives acknowledgement of receipt of a product/piece of product before sending the next product/piece of product. Traceroutes to your machine, rcz006.ust.hk show that the round trip time from Boulder is on the order of a quarter of a second: Matt's traceroute [v0.44] laraine.unidata.ucar.edu Mon Nov 18 10:13:43 2002 Keys: D - Display mode R - Restart statistics Q - Quit Packets Pings Hostname %Loss Rcv Snt Last Best Avg Worst 1. flra-n140.unidata.ucar.edu 0% 540 540 0 0 0 10 2. gin-n243-80.ucar.edu 0% 539 539 0 0 0 10 3. frgp-gw-1.ucar.edu 0% 539 539 1 1 2 14 4. 188.8.131.52 0% 539 539 1 1 3 17 5. 184.108.40.206 0% 539 539 2 1 2 11 6. kscy-dnvr.abilene.ucaid.edu 0% 539 539 14 12 13 60 7. kscyng-kscy.abilene.ucaid.edu 0% 539 539 12 12 13 32 8. iplsng-kscyng.abilene.ucaid.edu 2% 533 539 47 21 86 332 9. ipls-iplsng.abilene.ucaid.edu 0% 539 539 21 21 22 52 10. 220.127.116.11 0% 539 539 262 261 264 471 11. 18.104.22.168 0% 539 539 263 263 264 322 12. 22.214.171.124 0% 539 539 264 263 269 551 13. 126.96.36.199 0% 538 539 269 263 281 649 14. internet-gw1.ust.hk 0% 538 539 272 265 286 630 15. rcz006.ust.hk 0% 538 539 267 265 285 585 This means that at most 4 products per second can be delivered to your machine on any one rpc.ldmd connection. The 'WMO' feed, which is the combination of IDS, DDS, PPS, and HRS (DDS + PPS == DDPLUS) has a LOT of products in it. There is virtually no way that all of these products can be delivered to your machine using a single rpc.ldmd connection. The number of products available in the IDS|DDPLUS feed alone will make it impossible for you to have a reliable feed unless you take steps to split your feed request into multiple pieces. The objective is to change your system and LDM configurations to cause use of multiple rpc.ldmd connections between your machine and your upstream feed site. The current LDM gathers all feed requests to individual machines into a single rpc.ldmd. To get around this "feature", you have to play some games. Here's how: <as 'root'> o edit your /etc/hosts file and create an entry for your upstream feed site that contains several aliases. Here is what I recommend: 188.8.131.52 iita.rap.ucar.edu iita1.rap.ucar.edu iita2.rap.ucar.edu iita3.rap.ucar.edu iita4.rap.ucar.edu iita5.rap.ucar.edu -- be mindful that a tab is required as white space between the machine IP and the first name; all other white space can be spaces -- This creates 4 aliases for the machine iita.rap.ucar.edu. We will use these aliases to split your data request into smaller pieces. Next, check your /etc/nsswitch.conf file to make sure that the /etc/hosts file will be used first when looking-up machine names. Here is how we have one of our RedHat Linux 7.3 systems setup: hosts: files dns nisplus nis This specifies the search order when doing host look-ups to: 1st) files -> /etc/hosts 2nd) dns -> use DNS lookup 3rd) nisplus -> use nisplus 4th) nis -> use nis You may choose to specify nisplus and/or nis before dns, but you should set 'files' to be first in the list. <as 'ldm'> Now that you have several aliases for your feed host, you can split your data requests. How this can be done varies. I will show you how I setup a machine in Belem, Brazil that has traceroutes virtually identical to yours to receive data: change: request WMO ".*" iita.rap.ucar.edu to: #request WMO ".*" iita.rap.ucar.edu request DDPLUS|IDS "^[^S]" iita.rap.ucar.edu request DDPLUS|IDS "^S[AR]" iita1.rap.ucar.edu request DDPLUS|IDS "^S[^AR]" iita2.rap.ucar.edu request HDS "^H.[I-P]... KWB[^K]" iita3.rap.ucar.edu request HDS "^H[A-Z][ABCD][A-Z][0-9][0-9] KWB." iita4.rap.ucar.edu request HDS ".*(ECMWF|EGRR)" iita5.rap.ucar.edu What this does is split the request for DDPLUS|IDS data (global observational data) roughly into thirds by number of products. The HDS request lines request just the global model output from the HDS feed. I imagine that this would be all that you will need, since the majority of model data in HDS covers only North America. After you make the changes to ~ldm/etc/ldmd.conf, you will need to stop and restart your LDM: ldmadmin stop <wait until all LDM processes have exited> ldmadmin start At this point (assuming that you didn't have any typos in /etc/hosts, /etc/nsswitch.conf, and ~ldm/etc/ldmd.conf), you should see 7 rpc.ldmds running on your machine where previously you would have seen only 2. This same technique for feed splitting has worked incredibly well for the machine in Brazil that we are feeding. OK, so the procedures listed above are UGLY. We are working on a new version of the LDM that gets rid of all of this ugliness. Since this version will not be generally available for some unknown time, the feed splitting technique is what you will need to do. With all of the above, I would like to encourage you to upgrade your LDM installation to 5.2.2. The main reason for this is that 5.2+ allows sites to report realtime statistics back to the Unidata Program Center (UPC). 5.2.2 is only available as source, so you would need to build it locally. LDM versions 5.2.0 and 5.2.1 have known memory leaks in the real time statistics reporting facility, so please install 5.2.2. Here is the quick and dirty for doing an LDM installation: <login as 'ldm'> cd ~ldm ftp ftp.unidata.ucar.edu <user> anonymous <pass> your_full_email_address cd pub/ldm5 binary get ldm-5.2.2.tar.Z quit zcat ldm-5.2.2.tar.Z | tar xvf - cd ldm-5.2.2/src ./configure make make install sudo make install_setuids cd ~/ldm-5.2.2/bin <edit ldmadmin and make sure that: o '$hostname = "@HOSTNAME@"' is replaced by '$hostname = "rcz006.ust.hk"' o the location for Perl on your system is correct o the LDM product queue size is what you want Once you have finished the build, you will need to stop the LDM if it is running; switch your runtime link; deleted your old LDM queue; remake a new LDM queue; add a line to the LDM ldmd.conf file to report real time statistics; and restart your LDM: cd ~ ldmadmin stop rm runtime ln -s ldm-5.2.2 runtime ldmadmin delqueue ldmadmin mkqueue <- will take varying amounts of time; be patient cd etc <edit ldmd.conf and add the line: exec "rtstats -h rtstats.unidata.ucar.edu" put this after the line that starts pqact. ldmadmin start Once we start receiving realtime statistics from your, you will be able to look at how well you are receiving data online: http://www.unidata.ucar.edu/staff/chiz/latency/stats Click on the 'Statistics by Host' link from the above page and look for your host, rcz006.ust.hk. Please let us know if you have any questions on the above procedures. Cheers, Tom Yoksas >From: address@hidden >Date: Tue, 19 Nov 2002 01:56:34 +0800 (CST) >Subject: Away from my email [Re: 20021116: using the LDM to feed a remote site >in Hong Kong (cont.)] >I will be out of town until 20th Nov. >Your email regarding >20021116: using the LDM to feed a remote site in Hong Kong (cont.) >will be read when I return.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.