[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #EFY-891334]: Questions about memory management and the LDM


> At UNAVCO we're using LDM as a transfer mechanism from our ftp-in
> servers to our archiving clusters.  Perl scripts call pqinsert to add
> products to the ldm.pq in our dmx then internal ldm clients subscribe
> to appropriate feeds to extract files as a stream to feed a perl script
> to store files into a directory for further processing by more perl
> scripts.  We're running into some issues with product transfers and
> I've been assigned the task of looking into the LDM transfer issues.
> I suspect it has to do with the efficiency of the processing, and as I'm
> learning about LDM I have some thoughts about improving it's performance
> by changing our configuration.  That and updating our software package
> which is incredibly old (4.6.5).

4.6.5? Wow! That is old.

LDM 6 is almost a different beast. I'd recommend reading its documentation very 

> One of the first things I have considered changing is the ldm.pq file
> size is set at 20G which seems large for the file sets we're transferring.
> The swap file on the machine (solaris 5.10) is only 4G and I think that it
> should be tuned to have a larger system swap than the 20G ldm.pq. The pq
> stats show that we're never using more than about a 10th of the slots, and
> the largest file we've seen is a 3.8G file sent as an uncompressed tar.

The LDM-6 product-queue is a memory-mapped file, so it doesn't use swap space 
(at least on a modern O/S). The rule-of-thumb for swap space is to have twice 
as much swap as physical memory.

The rule-of-thumb for the size of the LDM product-queue is to make it large 
enough and with enough slots so that the minimum residence time of a 
data-product is at least one hour.

> I would like to tune the ldm.pq size down to 2G and I was wondering if
> the ldm.pq were down sized to fit in the swap space at a 2G file size
> what would happen if pqinsert were used to try and add a 3.8G file?

LDM-4's pqinsert(1) would likely crash. LDM-6's will error-exit.

> I'm confident that this could help us out with our transfer issues but
> I would like to know if others with more experience using LDM have any
> more practical advice.  I'm still in the middle of mapping our product
> distribution and when completed I think I can eliminate most duplicate
> requests though I believe if we updated our software LDM does a good
> job of taking care of that for me.

The LDM won't insert a product into the queue if a product with the same 
signature (i.e., MD5 checksum) is already in the queue.

> Any feedback or recommendations you could provide would be greatly
> appreciated.

It would have been good if you had attended our LDM training workshop. The next 
one will be next summer.

Feel free to drop by. I often work from home, so you should call ahead.

We could also use Google Hangouts.

> Thanks
> -Mike Rost
> Software Engineer I
> 6350 Nautilus Dr
> Boulder CO 80301
> 303-381-7467

Steve Emmerson

Ticket Details
Ticket ID: EFY-891334
Department: Support LDM
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.