Re: [gembud] Generating NEXRAD radar imagery with gpmap_gf

Look for a large .history file in the home directory of the userid that runs 
the scripts.  Sometimes this file can become huge and becomes a big memory hog 
as it is read in each time a shell script is launched.
If this is the case, delete it, and modify your .profile or .cshrc so a history 
file does not get generated except in interactive shell sessions.

--Kevin
________________________________________
From: gembud-bounces@xxxxxxxxxxxxxxxx [gembud-bounces@xxxxxxxxxxxxxxxx] on 
behalf of Greg Stossmeister [gstoss@xxxxxxxx]
Sent: Monday, April 30, 2012 3:28 PM
To: gembud@xxxxxxxxxxxxxxxx
Subject: Re: [gembud] Generating NEXRAD radar imagery with gpmap_gf

Daryl,
     No each process creates a temporary working subdirectory to run in based 
on the process id.

Greg

On Apr 30, 2012, at 1:23 PM, daryl herzmann wrote:

> Greg,
>
> Are all these processes running out of the same CWD (directory)?  Try 
> creating temp directories for each process and run the code from those 
> directories.
>
> daryl
>
> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>
>> Daryl,
>>   I have several shell scripts that I'm running out of cron every 5 minutes. 
>> Each shell script runs 10 gpmap_gf processes in sequence. I've tried running 
>> 1 - 6 scripts at a time. This typically works fine during the day with one 
>> of the these scripts completing in about 2 minutes. As evening comes on they 
>> take longer and longer to run and it seems like that take more and more 
>> memory. From the "top" command the scripts often use 500-800 MB of memory 
>> but in the evening this seems to mushroom to > 3GB per script. The load on 
>> the machine at night from these scripts alone jumps to >30  and by morning 
>> the machine usually dies with out of memory errors even though I'm 
>> automatically killing the scripts when they run longer than 2 minutes.
>>
>> Looking at /var/log/debug.log I'm seeing segfault errors:
>>
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2164]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff8d981128 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic abrt[2179]: saved core dump of pid 2164 
>> (/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf) to 
>> /var/spool/abrt/ccpp-201
>> 2-04-26-17:07:25-2164.new/coredump (827392 bytes)
>> Apr 26 17:07:25 sferic abrtd: Directory 'ccpp-2012-04-26-17:07:25-2164' 
>> creation detected
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2239]: segfault at 0 ip 
>> 000000392692ff7f sp 00007ffffd173658 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2242]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff8f4df6f8 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2247]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff73574d18 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2261]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff8bda1358 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2245]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff71495a28 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: Pid 2245(gpmap_gf) over core_pipe_limit
>> Apr 26 17:07:25 sferic kernel: Skipping core dump
>> Apr 26 17:07:25 sferic abrt[2260]: not dumping repeating crash in 
>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>> Apr 26 17:07:25 sferic abrt[2279]: not dumping repeating crash in 
>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2289]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fffca7118a8 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2286]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fffef00ac98 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2303]: segfault at 0 ip 
>> 000000392692ff7f sp 00007fff92019618 error 4 in 
>> libc-2.12.so[3926800000+186000]
>> Apr 26 17:07:25 sferic kernel: Pid 2303(gpmap_gf) over core_pipe_limit
>>
>> Greg
>>
>> On Apr 30, 2012, at 12:10 PM, daryl herzmann wrote:
>>
>>> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>>>
>>>> Does anyone generate a lot of individual NEXRAD level III products with 
>>>> gpmap_gf? I'm trying to generate real-time plots of NOQ Reflectivity and 
>>>> NOU Velocity from 30 radars in the midwest and its crashing my server 
>>>> after a few hours, even when I only run 3 plots at a time. I'm running 
>>>> GEMPAK6.4.0 on a RHEL 6 machine with 64 GB of memory. I'm wondering what 
>>>> I'm doing wrong and if someone has a better way of doing this.
>>>
>>> crashing your server, how?  Exhausting memory?  kernic panic?  Are the 
>>> processes not going away once running them?  How are you running them?
>>>
>>> daryl
>>>
>>> --
>>> /**
>>> * Daryl Herzmann
>>> * Assistant Scientist -- Iowa Environmental Mesonet
>>> * http://mesonet.agron.iastate.edu
>>> */
>>
>> -----------------------------------------------------------
>> Greg Stossmeister
>> Head - Collaborative Tools and Metadata Group
>> NCAR/Earth Observing Laboratory (EOL)
>> Telephone: (303)497-8692  Facsimilie:(303)497-2044
>> e-mail: gstoss@xxxxxxxx
>> Mailing Address:          Shipping Address:
>> NCAR/EOL                    UCAR Foothills Lab 1
>> P.O. Box 3000               3450 Mitchell Ln
>> Boulder, CO 80307      Boulder, CO 80301
>> ------------------------------------------------------------
>>
>>
>>
>>
>>
>> _______________________________________________
>> gembud mailing list
>> gembud@xxxxxxxxxxxxxxxx
>> For list information or to unsubscribe,  visit: 
>> http://www.unidata.ucar.edu/mailing_lists/
>>
>
> --
> /**
> * Daryl Herzmann
> * Assistant Scientist -- Iowa Environmental Mesonet
> * http://mesonet.agron.iastate.edu
> */

-----------------------------------------------------------
Greg Stossmeister
Head - Collaborative Tools and Metadata Group
NCAR/Earth Observing Laboratory (EOL)
Telephone: (303)497-8692  Facsimilie:(303)497-2044
e-mail: gstoss@xxxxxxxx
Mailing Address:          Shipping Address:
NCAR/EOL                    UCAR Foothills Lab 1
P.O. Box 3000               3450 Mitchell Ln
Boulder, CO 80307      Boulder, CO 80301
------------------------------------------------------------





_______________________________________________
gembud mailing list
gembud@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
http://www.unidata.ucar.edu/mailing_lists/