Paul, > Here's a simple depiction of the process: > > GFS files downloaded -> cf1 ldm <-> arps ldm <-> firewall <-> ldad ldm > and processed w/ wgrib2 > > 1. use wgrib2 to create *.grb2 files. > 2. insert *.grb2 files into cf1 queue > 3. arps ldm requests *.grb2 files from cf1 > 4. ldad ldm requests *.grb2 files from arps Thanks for clearing that up. The files are arriving at Arps in a timely manner but arrive at Ldad in an increasingly delayed manner, while other (presumably smaller?) files are not delayed. Is this correct? > My initial suspicions were a bottle neck at the firewall. However it is only > these grb2 files that see the delay, other data through the firewall to the > ldad ldm doesn't experience this issue. I will admit however that the grb2 > files are written to the arps queue approximately every two minutes though at > the one hour point on the ldad ldm side when I stop seeing the data come > through I have verified the files are still in the arps queue. The ldm > transactions between cf1 and arps are nearly instantaneous and never delayed > between the two. Are the other, timely files on a different LDM connection (i.e., due to a different REQUEST entry in the LDM configuration-file on Ldad) and do they use less bandwidth than the problematical files? If so, then there might be "packet shaping" occurring at the firewall. This is where the firewall preferentially throttles-down high-bandwidth TCP connections. We've seen this before at several institutions. We've seen that it can be difficult to 1) get the IT department to admit to such; and 2) get them to stop it. Is the Ldad LDM reporting rtstats(1) statistics? Are you plotting them? Do the plots show that high-volume connections have large latencies while low-volume ones don't? > I did increase the ldad queue size from the default size of 800M to 2G, but > the magic number still seems to be 1 hour. The LDM system has a "maximum latency" threshold, which is one hour by default. If a data-product arrives more than this threshold amount of time after it was created, then it will be discarded. Based on your description, this appears to be happening at the Ldad LDM due to the increasing latency. [Regarding sending a SIGUSR2 to the pqact(1) process:] > How do I do this? Set in a conf file then restart ldm or just execute from > terminal window? You can get the process ID of the pqact(1) process via the ps(1) utility or by searching the LDM log file for the string "pqact". Once you have it, you can use the kill(1) utility to send it signals. > Is the product-queue on some sort of RAID? > > No. I also looked on the MIDDS side and the arps queue was 6.5G (default 2G) > and the cf1 queue was 1.5G (default 1G). By "default 2G" do you mean that output from the command "ldmadmin config" on Arps shows that the size of the product-queue is set to "2G"? My question about the size of the computer (32 or 64 bit) should have been directed at Ldad and not the MIDDS systems. Regards, Steve Emmerson Ticket Details =================== Ticket ID: NGQ-500529 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.