>From: address@hidden >Organization: ULM >Keywords: 200306161954.h5GJs2Ld016710 LDM-6 IDD Hi Adam, >Just catching up on what you guys might of found. >From what i see tornado i feeding well with little to no latency. Just let >me know if you guys have found anything. The list of tests that have been run, and the interactions with contacts at LSU are longer than can be accurately recounted in a simple email, but I can say that a number of things have been investigated and several problems found and fixed. You can review all of the email transactions between Unidata and LSU by doing a search of our inquriy tracking system: http://www.unidata.ucar.edu/glimpsedocs/ghiddsupport.html Use 200306161954.h5GJs2Ld016710 as the search key and set the number of matches up to at least 40 (the default is 20). This will return a list of all email sent between UPC support and the LSU LDM contact in reverse chronological order. A very quick overview of what you will find follows: ULM: - needed upgrade to LDM-6 along with "tuning" of requests in ~ldm/etc/ldmd.conf. Tuning consisted of splitting feed requests and documentation. - LDM-6's ability to transfer data faster to receiving clients caused the ULM machine, tornado, to exhibit overheating-related system shutdowns. Investigations revealed that one of the case fans in the dual 900 Mhz PII Dell was not operating and was probably causing the over heating problem - after the overheating problem was overcome by opening tornado's case, real time statistics reported to Unidata by tornado for the HDS feed from seistan revealed a problem with the Internet-2 connection from ULM. Rerouting the connection to I1 (presumably the commodity internet), showed that there was an improvement in feeding IDD data but that a problem still existed. The HDS stream exhibited the feed problems showing latencies up to 6000 seconds even while feeds of the IDS|DDPLUS, UNIWISC, and FSL2 feeds showed acceptable, but still elevated levels. Switching the HDS feed from seistan to first emo.unidata.ucar.edu and then rainbow.al.noaa.gov showed that ULM could reliably get the HDS feed with very low latencies. This conclusively demonstrated that the feed problems being experienced by ULM were being caused by problems outside of the ULM domain. This, of course, was the conclusion that ULM had come to before contacting Unidata. LSU: - after verifying that the ULM feed problem was not related to anything at ULM, investigation moved to the srcc.lsu.edu, lsu.edu, and LANET domains. - extensive tests feeding the HDS stream to the LSU IDD relay node, seistan.srcc.lsu.edu from a machine in the unidata.ucar.edu domain, and then back to a different machine in the unidata.ucar.edu domain demonstrated that data could be fed with little to no latence from Unidata to LSU, but latencies for the same stream from LSU back to Unidata showed the same pattern exhibited at ULM. This finding was confirmed by feed tests from LSU to the University of South Florida in Tampa. - the system configuation of the LSU IDD relay node was examined in detail by Unidata support and systems staff, and, while the firewall rules in place were not efficiently organized, no show stoppers were detected. - conversations with LSU/SRCC support staff indicated that there was no "packet shaping" being done by the SRCC. Inquiry as to whether LSU telecommunications was running "packet shaping" got a negative reply. - LSU telecommunications was contacted to see if there was any known problems with networking equipment (e.g., routers, etc.) on campus. Since there were none, LSU telecommunications personnel contacted LANET personnel to see if they knew of any problems in their operations. They commented that there had been an open trouble ticket for "quite some time" in which CRC and retransmission errors were being seen. - last Friday, June 27, we noticed a significant change in the latencies being seen in the HDS feed from LSU to Unidata. For extended periods of time, the latencies dropped to near zero but spikes in the latencies were still being seen. A return to HDS high latencies was seen on Sunday afternoon/evening. - contact to SRCC and LSU personnel showed that no changes to either the srcc and LSU domains had been made over the weekend. In the early morning hours on Monday, the latencies for feeds eminating from LSU dropped to near-zero values and have remained there ever since. - a conference call between LSU/SRCC, LSU telecommunications, and Unidata representatives on Tuesday afternoon resulted general agreement that further troubleshooting was needed so that we will know exactly what was wrong and what was done to fix the problem. We are currently doing stress testing of the IDD node in the srcc.lsu.edu doman by having it feed a variety of IDD streams to multiple of downstream machines. The list of the machines and the streams they are ingesting from seistan.srcc.lsu.edu currently stands at: Machine Feeds from seistan ------------------------------+--------------------------------------------- emo.unidata.ucar.edu HDS chevy.unidata.ucar.edu HDS, IDS|DDPLUS, UNIWISC newshemp.unidata.ucar.edu HDS zasu.unidata.ucar.edu HDS, IDS|DDPLUS, UNIWISC zero.unidata.ucar.edu HDS, IDS|DDPLUS, UNIWISC imogene.unidata.ucar.edu HDS, IDS|DDPLUS, UNIWISC, FSL2 tornado.geos.ulm.edu HDS, FSL2, IDS|DDPLUS, NNEXRAD, UNIWISC hail.jsums.edu NNEXRAD, FNEXRAD, UNIWISC, IDS|DDPLUS aqua.nsstc.uah.edu UNIDATA (which is HDS, UNIWISC, IDS|DDPLUS) - During the conference call, we found out that LSU _does_ do packet shaping, but typically for traffice eminating from the student side of their network (to limit things like MP3 down/up loads). Given this, LSU telecommunications setup a separate, unimpeded channel for LDM traffic out of srcc.lsu.edu The newly created pipe has a limit set at 20 Mbps (!), so it is _highly_ unlikely that any IDD traffic that srcc.lsu.edu could produce would put even a noticible dent in the channel's capacity. At the moment, seistan is feeding 27 downstream LDM connections and receiving 8 feeds from upstream LDMs, for a total of 35 LDM connections. During the peak volume times for the HDS datastream, seistan should be relaying over 2.4 GB of data per hour. Additional downstream feed hosts will be added to the stress testbed if needed and if seistan can carry the load. The LSU campus network, which has an OC-3 connection to LANET, is showing little signs of the increased data volumes from seistan. It is entirely possible that the cause and fix for the feed problems that was being experienced by sites downstream of srcc.lsu.edu will never be found; there are simply too many networking organizations along the network path to be positive of where the failure lay. Our testing of srcc.lsu.edu is designed to stress the network as much as we can, but we may not be able to have seistan, which is a dual 400 Mhz PII Linux box, relay enough data to make a noticable impact in the network path. The short comment is that data being fed out of srcc.lsu.edu is now flowing with little to no latencies. Whether or not these latencies will remain low remains to be seen. If a return to high latencies is seen, Unidata, SRCC, and LSU telecommunications personnel will be actively trying to figure out where the problems are occurring and what steps need to be taken to solve them. Well, there you have it. Again, the full set of email transactions can be found in our inquiry tracking system. Cheers, Tom From address@hidden Sat Jul 5 20:21:33 2003 Subject: Thanks Unidata, From all of us here at ULM we thank everyone involved for the effort put forth to try and fix this problem. The passwords will stay the way they are for now on tornado until someone from unidata tells me they are done and i will change the passwords back to what they were. If you ever need them again just email me. oh, hehe one more thing. "I1" is the commodity Internet. I throught that i had said that awhile ago but i guess i was mistaken. Sorry for the confusion. If you ever have any questions feel free to email me. Also, if you need me to allow pinging on tornado i will. Thanks Adam Taylor Compuing Center University of Louisiana at Monroe
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.