Hi Joey, > NCEP/NCO is in the process of evaluating the ldm for sending large > quantities of data to NCDC for archival purposes. Folks have > recommended that I receive input from both of you as you have a great > deal of experience in dealing with the ldm software. > > Do either of you have a feel on what the limits are for data processing > via the ldm? Is there a line in the sand that should not be crossed? > Are there issues with large files as compared to small files? At what > data extreme would you not feel comfortable using the ldm? What are the > limiting factors? (Memory, hard drive space...etc) > > Thanks for you input. > > Joey We've received this question a number of times. We can tell you some general things to consider and provide you with some examples. Probably the most critical aspect of the LDM system is the size of the product-queue and the amount of physical memory. Ideally, you should have enough physical memory to hold the product-queue in memory all the time. This means that the amount of physical memory should be at least 30% greater than the size of the product-queue (with more memory being better, of course). The minimum size of the product-queue can be determined by multiplying the maximum rate at which data arrives by the minimum time that data should be available in the product-queue (typically one hour). Note that some systems cannot support product-queues larger than about 2 or 4 gigabytes because either the "off_t" or "size_t" data-type is a 32 bit integer, respectively. Processing data-products on the same system that receives them can slow things down considerably due to contention for the CPU. Consequently, the topology of an LDM network can greatly affect performance. One of the things that we recommend for systems that do a lot of processing is to either have multiple systems each receiving and processing a subset of the data or have a single system that receives all data and multiple, other systems that each process a subset of the data that they request from the front-end system. Each LDM child process that receives data executes an exclusive write-lock on the product-queue in order to store the data. Consequently, the more REQUEST entries that you have in the LDM configuration-file, the more contention will exist for writing into the product-queue. Interestingly enough, this tendency towards poorer performance with increased number of REQUEST entries is compensated, more or less, by multiple TCP connections between the same hosts having higher net throughput than a single TCP connection (depending on the details of the TCP implementation). Consequently, exactly how many REQUEST entries are too many depends on your exact circumstances (see the examples, below). And, of course, no LDM can receive data faster than the usable bandwidth of its network connection. Regarding the size of data-products: the LDM can handle products ranging from 1 byte to about 2 gigabytes equally easily. I would set the size of a data-product based on some natural division within the data-product itself rather than on some arbitrary value (remember, each data-product must have a unique, textual product-identifier). For example, a natural division for NEXRAD Level-II radar data is a single, multi-parameter horizontal sweep. And now for some examples. Our quasi-operational system at the Unidata Program Center is described at http://www.unidata.ucar.edu/committees/usercom/200604meeting/ldm-idd-status.html under the heading "IDD Relay Cluster idd.unidata.ucar.edu" (follow the links for more information). Currently, the two "accumulator" systems receive about 3.3 gigabytes of data per hour in about 155,000 data-products split over 18 or 19 REQUEST entries. The outbound data-rate has been stress-tested to about 76 gigabytes per hour, aggregate, over about 270 connections. The LDM on host ultrazone.ucar.edu, which is part of the TIGGE project <http://tigge.ucar.edu>, currently receives about 10 gigabytes per hour on 71 connections. The host is a SUN SPARC Sun-Fire-V890 with, I think, 18 GB of memory, a 14 GB product-queue and running SunOS 5.10. If I may ask, how much data are you planning on transmitting, on what sort of network, and with what sort of hardware budget? Regards, Steve Emmerson Ticket Details =================== Ticket ID: SYM-524686 Department: Support LDM Priority: Normal Status: Closed
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.