[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #OJL-403914]: LDM Memory Issues

Justin,

> You have a good memory :-) .  Yes, you helped our ESA with a similar
> problem on a "virtual" machine a month or so ago and I guess you all
> arrived at that conclusion...that there is a problem with virtualization.
> 
> Anyhow, I can't answer whether these "cloud" systems are real or
> virtual.  Supposedly, these servers use pooled resources for disk space,
> processor cores, memory!!!, etc., and therefore, can be increased or
> decreased as needed.  Sounds good in writing...not sure how good it
> actually works and therein may be the problem.
> 
> However, this may not be a problem after all.  I misspoke somewhat in my
> email in that I just assumed that as the memory utilization by
> ldmd/pqact increased, eventually the machine would crash, because that
> what the "virtual" machine did.  However, after composing that email
> over the weekend, we decided to test the theory and actually send data
> to it to see what would happen.  We are currently ingesting several
> NEXRAD3 sites from Vortex, some NEXRAD2 sites from Texas Tech and the
> entire text feed from Vortex.  Anyhow, when free memory gets very low
> (8-16 MB), the memory increases stop and the machine continues to run
> fine (at least that's what we have seen over the last several hours).
> The machine has used some swap space, but very little.
> 
> Feel free to pop in and look around, but maybe I was wasting your time
> and things will operate normally??

I logged onto the system in question and looked around but didn't see anything 
untoward. (Didn't look very long, though.)

You should know that Linux systems allow processes to accumulate memory up to 
the limit of physical memory before resorting to use of swap space. Also, LDM 
processes dynamically allocate memory for the data-product buffer based on to 
the largest data-product seen, so you should expect memory usage to increase 
until the largest data-product is seen. These two factors might be the basis 
for your "memory leak".

I modified your crontab(1) table: I added entries for accumulating LDM and 
system metrics and for purging the metrics files. I suggest you install the 
gnuplot(1) package and then use the "ldmadmin plotmetrics" command to visualize 
the health of your system.

> XXXXXX.com
> user: XXXXXX
> pass: XXXXXX
> 
> Thanks, Steve, always appreciate your insight.
> 
> Justin
> 
> "You can easily judge the character of a man by how he treats those who can 
> do nothing for him."
> -James D. Miles
> 
> Justin Weaver
> Meteorologist-in-Charge
> National Weather Service
> US Government/DOC/NOAA
> Lubbock, Texas
> 806.745.3916 x222
> weather.gov/lubbock
> 
> Disclaimer:  The views expressed in this email are my own and do not
> necessarily reflect those of the National Weather Service, NOAA,
> Department of Commerce nor the Executive Branch of the Federal
> Government.


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: OJL-403914
Department: Support LDM
Priority: Normal
Status: Closed