Hi Greg (with CC to Sen), Two of us (Steve Emmerson and I) have been working with Sen to get a better understanding of the LDM data feeds and processing on titan.met.sjsu.edu, and to do some tuning to help titan function better. A lot has been learned over the past week, and Sen asked that we update you on what has been found and changed, and the effect it has had on titan's performance. First, what has been done - increase the size of the LDM queue from 500 MB (which is the default) to 12 GB The first thing that was found on titan was the LDM queue was not sized properly to handle the volume of data that titan was/is REQUESTing. Our "rule of thumb" is to try and size one's LDM queue so that it will hold about 1 hour of received data. The principle reason for this is that the LDM can reject newly received products if they have the same MD5 signature as ones already in the LDM queue. Rejection of duplicate products decreases the number/volume of products that get processed by LDM pattern-action file actions. A quick look at the real time stats that titan has been sending back to us (links are included below) showed that the volume of data being written into titan's LDM queue was peaking at over 100 GB per hour. This anomalous volume was a BIG red flag for us since the feeds being REQUESTed do not contain this much data. The cause of the anomalous volume of data being inserted into titan's LDM queue was that duplicate products were not being rejected as the product residency time in the LDM queue was on the order of a few seconds, not the recommended one hour. The extremely low residency time resulted in some products that had been received not being processed at all, and in a LOT of products that had been received and processed being received and processed again. - we find it most useful for sites running the LDM to monitor system performance by running the LDM utility 'ldmadmin addmetrics' once per minute from a cron job Since this was not being done on titan, it was added. The log file used for performance monitoring is ~ldm/logs/metrics.log. This file will be "rotated" (renamed to metrics.log.1, etc.) periodically by another action in 'ldm's crontab. One of the things that the metrics.txt log file contains is the age of the oldest product in the LDM queue. It was by turning on generation of the metrics.log file that the extremely short residency time in the LDM queue was found. Again, when metrics monitoring was first turned on, the residency time (which is indicated by the age of the oldest product in the queue) was found to be only a few seconds, not an hour like is recommended. Steve and my hunch was that the anomalously high volume of data flowing into titan's LDM queue was a result of products being received and inserted into the LDM queue multiple times. This can easily happen when there are redundant REQUESTs for feeds. In order to increase product residency times, titan's LDM queue was increased from 500 MB to 12 GB. It was felt that making the queue any larger would interfere with the processing that is being done on titan since titan only has 24 GB of RAM. After increasing the LDM queue size, we saw a significant decrease in the volume of data received being reported in the real time stats that titan is sending to us. We interpret this as an indication that rejection of products received more than once is now working as designed. The decrease in data being received has also resulted in a decrease in the amount of processing (decoding) that titan is doing since it no longer is running the same actions on duplicate data. - eliminate duplicate LDM feed REQUESTs while making sure that all of the data that was being REQUESTed is still being REQUESTed It turns out that there was quite a bit of duplication of REQUESTs for a number of the data feeds that titan was asking for. The duplicated feed REQUESTs were eliminated, and this resulted in a further decrease in not only the data being received by titan, but in the amount of LDM-related processing that titan is doing (e.g., decoding data into GEMPAK-compatible formats, etc.). - consolidated feed REQUESTs After letting the LDM run for awhile after increasing the size of the LDM queue and eliminating duplicate feed REQUESTs and while monitoring the reported reception latencies for the various feeds, efforts turned to consolidating feed REQUESTs where consolidation made sense. This was done to decrease the number of LDM processes that run continuously. Keeping down the number of feed REQUESTs that a system makes will help to decrease the LDM related processing load as there will be fewer REQUESTing processes. At the same time, splitting feed REQUESTs for high volume feeds like what is being done for CONDUIT has the beneficial effect of minimizing reception latencies. Exactly how one should consolidate some feed REQUESTs while splitting others is totally dependent on the number of products and volumes in the feeds being REQUESTed. Smart splitting of feed REQUESTs is also dependent on the LDM/IDD Product IDs for the products in a feed. Each CONDUIT product, for instance, has a sequence number as the last value in the Product ID. This sequence number makes it easy to split the feed in a way that all data is REQUESTed. The HRRR products in the FSL2 feed that originates at NOAA/GSD, on the other hand, are not easily split into multiple, disjoint REQUESTs. The other feeds that suffer from the same problem as FSL2 are NGRID, FNMOC and HDS, and this is a problem for NGRID and FNMOC as they are both high volume feeds that have lots of products. NEXRAD2, which contains NEXRAD Level 2 volume scan chunks, is a bit easier to split as is NEXRAD3, which contains NEXRAD Level 3 products. NEXRAD2 is one of the higher volume feeds in the IDD, and NEXRAD3 is the feed that has the most number of products per hour. - installed the latest LDM release, ldm-6.13.6 The latest release of the LDM was installed to make sure that none of what is being seen is a result of an older and possibly less efficient LDM. It was not expected that this would result in significant improvements in terms of getting and processing data, and it did not. Nonetheless, it is always the best idea to be running the latest version of any software since newer releases typically are more efficient and many times have new features that can be useful. Where titan stands now: - titan is now processing a LOT more of the data that it is REQUESTing than it was before the changes outlined above Also, the processing impact on the file system as measured by I/O wait has decreased substantially. This is a very good thing since the multiple processing of data that had already had been received was likely causing performance problems indicated by very high I/O waits that you reported in a previous email. - splitting high volume feeds to minimize their product latencies An attempt was made to split the single FSL2 HRRR feed REQUEST into thirds with the hope that this would help reduce the receipt latency for those HRRR products. The latencies for the FSL2 HRRR products have decreased a bit, but not as nicely/substantially/much as the latencies for the CONDUIT feed. The reason for this is that the FSL2 HRRR products Product IDs do not lend themselves to an easy split of feed REQUEST(s). In practice this means that one needs to have sufficient network bandwidth to receive the high volume FSL2 feed in a single feed REQUEST. The latencies for NGRID products remain high enough that not all of the NGRID products are actually being received, since their residency time in the upstream LDM queue(s) is not larger than the time it takes for the products to be received on titan. The problem with the NGRID feed is exactly analogous to that with the FSL2 feed. FNMOC latencies also remain high, but it appears that all of the products are received and processed for most hours. NEXRAD2 latencies also remain very high. In fact, a lot of the time NEXRAD2 products are not being received since the latencies exceed their residency time in the upstream LDM's queue. This is a problem that may be mitigated (but possibly not solved) by smart splitting of the NEXRAD2 feed REQUEST into multiple, disjoint REQUESTs. Observations: - After the experimentation with consolidating and splitting feed REQUESTs, it is our opinion that the principle cause of the inability to receive all of the data being REQUESTed is the network bandwidth available to titan We say this after reviewing a couple plots of the metrics being gathered by 'ldmadmin plotmetrics'. We encourage you to run 'ldmadmin plotmetrics' as user 'ldm' to get an idea of the variety of metrics that are now being gathered once per minute. In particular, we encourage you to look at: - the plot that shows the time series of CPU modes and I/O wait This plot shows that titan is idle much of the time, so the machine is not the cause of data not being processed. - the plot that shows the time series of CPU load NB: titan has 24 CPUs, so load averages of up to 10 should not be interpreted as being overly high. Our interpretation of this plot is that titan does get busy, but not criplingly so. Our view is also based on a careful review of the real time stats latencies being reported by titan back to us. You can generate current plots of latencies for all feeds being requested online at: Unidata HomePage http://www.unidata.ucar.edu Data -> IDD Operational Status http://rtstats.unidata.ucar.edu/rtstats/ Statistics by Host http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex titan.met.sjsu.edu [6.13.6] http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/siteindex?titan.met.sjsu.edu - further increasing the size of the LDM queue could result in even fewer products being redundantly received (i.e., more duplicates being rejected), and this would, in turn, reduce the processing load even more NB: The only way that the LDM queue could/should be increased to something like the maximum amount of data being received in an hour is by increasing the amount of RAM installed in titan. The following cumulative volume summary listing from titan shows what is currently being received by titan: http://rtstats.unidata.ucar.edu/cgi-bin/rtstats/rtstats_summary_volume?titan.met.sjsu.edu Data Volume Summary for titan.met.sjsu.edu Maximum hourly volume 60026.065 M bytes/hour Average hourly volume 34232.870 M bytes/hour Average products per hour 387841 prods/hour Feed Average Maximum Products (M byte/hour) (M byte/hour) number/hour FSL2 11177.067 [ 32.650%] 17054.068 15580.956 CONDUIT 7957.826 [ 23.246%] 21930.253 89129.044 NGRID 4936.321 [ 14.420%] 12914.648 32253.000 NEXRAD2 3070.162 [ 8.968%] 11036.379 33935.333 NEXRAD3 2916.793 [ 8.520%] 3480.750 122637.156 FNMOC 1968.504 [ 5.750%] 10018.393 3435.956 HDS 1190.873 [ 3.479%] 1848.008 43404.244 FSL3 608.333 [ 1.777%] 644.139 52.911 FNEXRAD 130.943 [ 0.383%] 155.958 104.156 NIMAGE 98.102 [ 0.287%] 146.058 124.044 UNIWISC 96.527 [ 0.282%] 143.653 49.422 IDS|DDPLUS 76.936 [ 0.225%] 95.393 47044.200 EXP 4.067 [ 0.012%] 17.561 31.400 LIGHTNING 0.416 [ 0.001%] 0.917 59.111 If there was enough memory (e.g., 96 GB or more), we would recommend that the LDM queue size be increased the to about 60 GB. If it is only possible to increase titan's RAM to 64 GB, the LDM queue size could be increased to 35 GB. Either of these would likely have beneficial effects. Our motto is "the more RAM the better" :-) - it was verified that the Ethernet interface (em1) on titan is running at 1 Gbps (1000 Mbps) (ethtool em1 | grep Speed) This was checked since we have seen some institutions running their Ethernet interface at 100 Mbps, and this is not fast enough to get all of the data desired. The fact that the Ethernet interface is being run at 1 Gbps tells us that the inability to get all of the FSL2 HRRR data in a more timely manner in a single REQUEST would not be solved by installing a higher speed (e.g., 10 Gbps) Ethernet interface, and that the limiting factor is likely the network bandwidth available to titan. Our recommendations: - install more RAM in titan and then - further increase the LDM queue size Exactly how much will depend on how much more RAM can be added to titan. - meet with SJSU network folks to see if there is any way more network resources could be made available to titan If the available network bandwidth is limited, then we recommend: - reducing the set of data being REQUESTed on titan Things to consider are: Are all of the products in the FSL2 HRRR feed being used? Are all of the products in the CONDUIT feed being used? Are all of the products in the NEXRAD2 and NEXRAD3 feeds being used? Are all of the products in the NGRID, FNMOC and HDS feeds being used? The volume of data in the other feeds being REQUESTed is relatively small, so there is not much to be gained by restricting what is REQUESTed in those feeds. Final comments: - we realize that the information contained above is pretty dense and so may be hard to wrap one's mind around all at once Please send any/all questions that occur to you, and we will try to be more clear. - titan is a decent machine titan can be used effectively by itself or in combination with new equipment to be purchased under the Unidata equipment grant that SJSU was awarded. - no "heroic" effort is needed to configure any new machine to be able to handle the data that is desired By "heroic", we mean that switching to use of SSDs is probably not needed. - one of the best expenditures of money when purchasing new or upgrading old equipment is to buy more RAM More RAM in titan would allow for the LDM queue size to be increased, and this should have the beneficial effect of decreasing the volume of data to be processed (by LDM rejection of duplicate products), and this, in turn, will lower the impact (e.g., I/O wait) on the file system. As a comparison, the machines we are running here in Unidata that receive and process ALL of the data available in the IDD are configured with a minumum of 192 GB of RAM. Those machines are also running the ZFS file system, but we believe that current implementations of XFS should also work very well. Cheers, Tom -- **************************************************************************** Unidata User Support UCAR Unidata Program (303) 497-8642 P.O. Box 3000 address@hidden Boulder, CO 80307 ---------------------------------------------------------------------------- Unidata HomePage http://www.unidata.ucar.edu **************************************************************************** Ticket Details =================== Ticket ID: MIW-275261 Department: Support LDM Priority: Normal Status: Closed =================== NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.