Re: [gembud] Generating NEXRAD radar imagery with gpmap_gf

Daryl,
    I noticed .history was bizarre last week - it had a ton of stuff in it that 
didn't look like the normal history stuff I was used to seeing. We deleted it 
this morning and I set my .cshrc to only save the last 40 commands.

Greg

On Apr 30, 2012, at 1:40 PM, daryl herzmann wrote:

> 
> Offline... How large is your ~/.history file?
> 
> daryl
> 
> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
> 
>> Daryl,
>>    No each process creates a temporary working subdirectory to run in based 
>> on the process id.
>> 
>> Greg
>> 
>> On Apr 30, 2012, at 1:23 PM, daryl herzmann wrote:
>> 
>>> Greg,
>>> 
>>> Are all these processes running out of the same CWD (directory)?  Try 
>>> creating temp directories for each process and run the code from those 
>>> directories.
>>> 
>>> daryl
>>> 
>>> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>>> 
>>>> Daryl,
>>>>  I have several shell scripts that I'm running out of cron every 5 
>>>> minutes. Each shell script runs 10 gpmap_gf processes in sequence. I've 
>>>> tried running 1 - 6 scripts at a time. This typically works fine during 
>>>> the day with one of the these scripts completing in about 2 minutes. As 
>>>> evening comes on they take longer and longer to run and it seems like that 
>>>> take more and more memory. From the "top" command the scripts often use 
>>>> 500-800 MB of memory but in the evening this seems to mushroom to > 3GB 
>>>> per script. The load on the machine at night from these scripts alone 
>>>> jumps to >30  and by morning the machine usually dies with out of memory 
>>>> errors even though I'm automatically killing the scripts when they run 
>>>> longer than 2 minutes.
>>>> 
>>>> Looking at /var/log/debug.log I'm seeing segfault errors:
>>>> 
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2164]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff8d981128 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic abrt[2179]: saved core dump of pid 2164 
>>>> (/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf) to 
>>>> /var/spool/abrt/ccpp-201
>>>> 2-04-26-17:07:25-2164.new/coredump (827392 bytes)
>>>> Apr 26 17:07:25 sferic abrtd: Directory 'ccpp-2012-04-26-17:07:25-2164' 
>>>> creation detected
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2239]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007ffffd173658 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2242]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff8f4df6f8 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2247]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff73574d18 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2261]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff8bda1358 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2245]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff71495a28 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: Pid 2245(gpmap_gf) over core_pipe_limit
>>>> Apr 26 17:07:25 sferic kernel: Skipping core dump
>>>> Apr 26 17:07:25 sferic abrt[2260]: not dumping repeating crash in 
>>>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>>>> Apr 26 17:07:25 sferic abrt[2279]: not dumping repeating crash in 
>>>> '/export/ldm/home/gempak/GEMPAK6.4.0/os/linux64/bin/gpmap_gf'
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2289]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fffca7118a8 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2286]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fffef00ac98 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: gpmap_gf[2303]: segfault at 0 ip 
>>>> 000000392692ff7f sp 00007fff92019618 error 4 in 
>>>> libc-2.12.so[3926800000+186000]
>>>> Apr 26 17:07:25 sferic kernel: Pid 2303(gpmap_gf) over core_pipe_limit
>>>> 
>>>> Greg
>>>> 
>>>> On Apr 30, 2012, at 12:10 PM, daryl herzmann wrote:
>>>> 
>>>>> On Mon, 30 Apr 2012, Greg Stossmeister wrote:
>>>>> 
>>>>>> Does anyone generate a lot of individual NEXRAD level III products with 
>>>>>> gpmap_gf? I'm trying to generate real-time plots of NOQ Reflectivity and 
>>>>>> NOU Velocity from 30 radars in the midwest and its crashing my server 
>>>>>> after a few hours, even when I only run 3 plots at a time. I'm running 
>>>>>> GEMPAK6.4.0 on a RHEL 6 machine with 64 GB of memory. I'm wondering what 
>>>>>> I'm doing wrong and if someone has a better way of doing this.
>>>>> 
>>>>> crashing your server, how?  Exhausting memory?  kernic panic?  Are the 
>>>>> processes not going away once running them?  How are you running them?
>>>>> 
>>>>> daryl
>>>>> 
>>>>> --
>>>>> /**
>>>>> * Daryl Herzmann
>>>>> * Assistant Scientist -- Iowa Environmental Mesonet
>>>>> * http://mesonet.agron.iastate.edu
>>>>> */
>>>>