[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #EFY-891334]: Questions about memory management and the LDM


> Thanks Steve: I appreciate the dialog.
> Iâll see about getting into a training workshop.  I only started here mid 
> July so Iâm still learning.
> There are many things here that havenât been updated in terms of software.  
> Thatâs what Iâm starting to make my way through is the entanglement of 
> old software.  I have been reading the documentation on ldm 4.6.12 and I hope 
> to download and build it next week.

I hope you're referring to LDM 6.12.6 rather than 4.6.12.

> Iâm debating right now if I should have the sysadmins bring the gcc and 
> perl up to date on the machines.  Iâm sure it would help as well, but 
> itâs a long process to get that done.  At any rate itâs one battle at a 
> time as I map out some milestones for progress.

The only necessity is that the perl interpreter be version 5.

> I still have some uncertainty about the behavior of ldm.  I was under the 
> impression that memory mapped files were mapped to the virtual memory of the 
> machine(physical and swap).  Part of what is perceived with other users here 
> in their experience with ldm is that it is a problematic process on a solaris 
> machine, because when it is running it takes up every bit of the physical 
> memory.  That observation is based on the file size we have configured, so 
> when running with other processes on the server there is plenty of swapping 
> going on.  I guess what youâre telling me does make sense though in the 
> fact that although all the physical memory is consumed, there is still swap 
> space left on the drive, so conceptually the ldm.pq file being memory mapped 
> is like extending the swap space of the OS.

Virtual memory is specific to a process rather than a machine. The LDM 
product-queue is memory-mapped into the virtual memory of each process that has 
it open.

> I wasnât getting that impression from reading the documentation online, it 
> seemed that the transfer was between the memory mapped file and the virtual 
> memory of the machine, so it made sense to have a swap space large enough to 
> augment the physical memory of the machine and expand the size of the virtual 
> memory to be large enough to put a 20G ldm.pq file into a mapping and still 
> have room left for the other OS processes.

Our advice is to have sufficient physical memory to hold the entire 
product-queue plus the O/S and its buffers. Multiply the size of the 
product-queue by 3/2 and you should be good.

> The reason we have a large file size for the queue is we would like to have 
> more than an hours worth of time for files.  Looking at our stats our mean 
> retention time right now is about five days.  Probably way too much as I 
> suspected so Iâll definitely consider trimming down the file size of ldm.pq 
> and see how that effects the performance and delivery.

5 days? Yeah, you can make it smaller.

> In the case of attempting to insert a file larger than the ldm.pq size does 
> that imply that LDM-4 will stop processing files through ldm until it is 
> restarted, and with an error thrown in LDM-6 will it continue to process or 
> will it stop ingesting files until restarted as well.

If the files are being inserted via pqinsert(1) *and* that process is running 
*inside* the process-group of the LDM server (by means of an EXEC entry in the 
LDM configuration-file) then the LDM-4 pqinsert(1) process will likely crash 
due to a SIGSEGV and take the entire LDM system down with it. An LDM-6 
pqinsert(1) processs in the LDM server's process group, however, will simply 
error-exit and the LDM system will stay up. If the pqinsert(1) process is not 
in the LDM server's process group, then neither LDM system will terminate. 
There will be a delay up to 30 seconds, however, before the inserted product is 
relayed downstream.

> The reason I would like to stay with the LDM is that it is in place, it 
> recognizes file duplication at the insertion point.  In my previous email I 
> was referring to downstream ldmâs that are requesting duplicated file sets 
> from the up stream.  I think if we can get our end to end path configurations 
> sorted out I believe we can tune the system to meet our needs.  Thanks for 
> the help and if I do get a chance Iâll see about dropping by UCAR after the 
> first of the year to get a better sense of how I could make ldm work for us.

You'd be welcome.

> Thanks
> -Mike

Steve Emmerson

Ticket Details
Ticket ID: EFY-891334
Department: Support LDM
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.