[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[GEMPAK #DXU-339106]: gempak stopped making images

Leonard,

Given that at one time you were able to run your image generation script
and has since stopped working, and a reboot would have cleared up any
old message queues, too may processes, out of memory problems etc., then
you may need to check your Xvfb virtual X display. I see your script is 
attampting to
launch "gf" which required an X server to connect to. When automating scripts, 
most users 
have Xvfb running as the display to connect to.Check to ensure that you have 
that running and
your DISPLAY environmental variable is set appropriately. Alternatively, try 
either
the "gif" driver to avoid the need for connecting to a display, or use 
"gdcntr_gf"
as a test to see if there is a problem using message queues to contact the gplt
process. Check the working directory of your script, and prossibly remove the 
gemglb.nts and 
last.nts files as they might have become corrupt.


Steve Chiswell
Unidata User Support



> I have tried to reboot, but the problem persists. I have stopped gempak for 
> now and our wrf simulations are still processing. I'll play with it a litte 
> more and see if I can find the problem.
> Thanks for your help.
> Leonard
> 
> 
> -----Original Message-----
> From: Unidata GEMPAK Support [mailto:address@hidden]
> Sent: Thu 7/5/2007 3:21 PM
> To: Leonard H. Montenegro
> Cc: address@hidden
> Subject: [GEMPAK #DXU-339106]: gempak stopped making images
> 
> Leonard,
> 
> if "kill pid" doesn't succeed in killing a process, then "kill -9 pid"
> should kill the process.
> 
> In your case, the zombie processes may still be trying to either talk to a 
> devide driver,
> such as "xw", "gf" or "ps", or waiting for a response from the message queue
> (the ipcs command will show message queues in use).
> 
> I cannot tell you at what point you ran out of resources. If your script is 
> unable to
> connect to a DISPLAY that it was using, then that would be a problem. 
> Otherwise, the use of cntl-C
> is bad since it doesn't remove the message queues.
> 
> After running "cleanup -c" if message queues still exist, and gplt and devices
> still exist, then a reboot is the simplest solution- but you still need to
> examine the running plot script to determine if the error condition still 
> exists.
> 
> Steve Chiswell
> Unidata User SUpport
> 
> 
> 
> 
> > I tried that but no luck.
> > I tried to kill all the gplt prcesses, but they won't stop. The I ran 
> > cleanup -c and re-ran the script, but it still had the same error. I used 
> > cntrl-C to kill my gempak process.
> >
> > wrf@localhost-> ps -ax | grep gplt
> > Warning: bad syntax, perhaps a bogus '-'? See 
> > /usr/share/doc/procps-3.2.6/FAQ
> > 5587 ?        Z      0:00 [gplt] <defunct>
> > 8679 ?        Z      0:00 [gplt] <defunct>
> > 9675 ?        Z      0:00 [gplt] <defunct>
> > 19504 ?        Z      0:00 [gplt] <defunct>
> > 25470 ?        Z      0:00 [gplt] <defunct>
> > 3158 ?        Z      0:00 [gplt] <defunct>
> > 4793 pts/1    S+     0:00 grep gplt
> > wrf@localhost-> kill 5587
> > wrf@localhost-> ps -ax | grep gplt
> > Warning: bad syntax, perhaps a bogus '-'? See 
> > /usr/share/doc/procps-3.2.6/FAQ
> > 5587 ?        Z      0:00 [gplt] <defunct>
> > 8679 ?        Z      0:00 [gplt] <defunct>
> > 9675 ?        Z      0:00 [gplt] <defunct>
> > 19504 ?        Z      0:00 [gplt] <defunct>
> > 25470 ?        Z      0:00 [gplt] <defunct>
> > 3158 ?        Z      0:00 [gplt] <defunct>
> > 4796 pts/1    S+     0:00 grep gplt
> > wrf@localhost->
> >
> >
> > wrf@localhost-> script_drvr.csh 
> > /home/wrf/wrf/runs/fortuna/wrfpost/gempak/2007070412_wrf_d01.gem
> >
> > Begin script_drvr.csh
> >
> > Creating process: gplt for queue 786432
> > GDATTIM   Grid date/time                    LAST
> > GLEVEL    Grid level                        500
> > GVCORD    Grid vertical coordinate          PRES
> > GFUNC     Scalar grid                       TMPC
> > GDFILE    Grid file                         $GEMDATA/HRCBOB.GRD
> > CINT      Contour interval/min/max          0
> > LINE      Color/type/width/label/smth/fltr  3
> > MAP       Map color/dash/width/filter flag  1
> > MSCALE    fgc;bgc;mask/units/lat;hide/valu  0
> > TITLE     Title color/line/title            1
> > DEVICE    Device|name|x size;y size|color   XW
> > SATFIL    Satellite image filename(s)
> > RADFIL    Radar image filename(s)
> > IMCBAR    Color/ornt/anch/x;y/ln;wd/freq
> > PROJ      Map projection/angles/margins|dr  MER
> > GAREA     Graphics area                     WV
> > IJSKIP    Iskp;Istrt;Istp/Jskp;Jstrt;Jstp
> > CLEAR     Clear screen flag                 YES
> > PANEL     Panel loc/color/dash/width/regn   0
> > TEXT      Size/fnt/wdth/brdr/N-rot/just/hw  1
> > SCALE     Scalar scale / vector scale       999
> > LATLON    Line color/dash/width/freq/inc/l
> > HILO      Color/symbol/rng/rad/cnt/intp
> > HLSYM     HILO txt size/posn/font/wdth/hw
> > CLRBAR    Color/ornt/anch/x;y/ln;wd/freq|t
> > CONTUR    Subbox/smooth                     0
> > SKIP      Skip_cntr/skip_plt_x;skip_plt_y   0
> > FINT      Fill interval/min/max             0
> > FLINE     Fill colors/fill types            10-20
> > CTYPE     Contour type:  C/F                C
> > LUTFIL    Enhancement lookup table filenam
> > STNPLT    Txtc/txt attr|marker attr|stnfil
> > Parameters requested: GDATTIM,GLEVEL,GVCORD,GFUNC,GDFILE,CINT,LINE,MAP,
> > MSCALE,TITLE,DEVICE,SATFIL,RADFIL,IMCBAR,PROJ,GAREA,IJSKIP,CLEAR,PANEL,TEXT,
> > SCALE,LATLON,HILO,HLSYM,CLRBAR,CONTUR,SKIP,FINT,FLINE,CTYPE,LUTFIL,STNPLT.
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> 
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> 
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> 
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> 
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> 
> > GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR> GEMPAK-GDCNTR>Creating 
> > process: gf for queue 819201
> > *** glibc detected *** gdcntr: double free or corruption (out): 
> > 0x000000000171aa20 ***
> > ======= Backtrace: =========
> > /lib64/libc.so.6[0x36fd96d7a3]
> > /lib64/libc.so.6(__libc_free+0x84)[0x36fd96d924]
> > gdcntr[0x48e62c]
> > gdcntr[0x48c37f]
> > gdcntr[0x48abba]
> > gdcntr[0x479507]
> > gdcntr[0x4788a7]
> > gdcntr[0x4782f5]
> > gdcntr[0x46f043]
> > gdcntr[0x42d01a]
> > gdcntr[0x42a511]
> > gdcntr[0x42a2a4]
> > gdcntr[0x436108]
> > gdcntr[0x4292f7]
> > gdcntr[0x422626]
> > gdcntr[0x406f7a]
> > gdcntr[0x404387]
> > gdcntr[0x4fa9c6]
> > /lib64/libc.so.6(__libc_start_main+0xf4)[0x36fd91d084]
> > gdcntr[0x403759]
> > ======= Memory map: ========
> > 00400000-00534000 r-xp 00000000 fd:00 3178607                            
> > /home/wrf/GEMPAK5.10.1/os/linux64/bin/gdcntr
> > 00633000-00639000 rw-p 00133000 fd:00 3178607                            
> > /home/wrf/GEMPAK5.10.1/os/linux64/bin/gdcntr
> > 00639000-0172b000 rw-p 00639000 00:00 0                                  
> > [heap]
> > 36fd700000-36fd719000 r-xp 00000000 fd:00 1179877                        
> > /lib64/ld-2.4.so
> > 36fd819000-36fd81a000 r--p 00019000 fd:00 1179877                        
> > /lib64/ld-2.4.so
> > 36fd81a000-36fd81b000 rw-p 0001a000 fd:00 1179877                        
> > /lib64/ld-2.4.so
> > 36fd900000-36fda3f000 r-xp 00000000 fd:00 1179878                        
> > /lib64/libc-2.4.so
> > 36fda3f000-36fdb3f000 ---p 0013f000 fd:00 1179878                        
> > /lib64/libc-2.4.so
> > 36fdb3f000-36fdb43000 r--p 0013f000 fd:00 1179878                        
> > /lib64/libc-2.4.so
> > 36fdb43000-36fdb44000 rw-p 00143000 fd:00 1179878                        
> > /lib64/libc-2.4.so
> > 36fdb44000-36fdb49000 rw-p 36fdb44000 00:00 0
> > 36fdc00000-36fdc80000 r-xp 00000000 fd:00 1179880                        
> > /lib64/libm-2.4.so
> > 36fdc80000-36fdd80000 ---p 00080000 fd:00 1179880                        
> > /lib64/libm-2.4.so
> > 36fdd80000-36fdd81000 r--p 00080000 fd:00 1179880                        
> > /lib64/libm-2.4.so
> > 36fdd81000-36fdd82000 rw-p 00081000 fd:00 1179880                        
> > /lib64/libm-2.4.so
> > 3703d00000-3703d0d000 r-xp 00000000 fd:00 1179881                        
> > /lib64/libgcc_s-4.1.0-20060304.so.1
> > 3703d0d000-3703e0d000 ---p 0000d000 fd:00 1179881                        
> > /lib64/libgcc_s-4.1.0-20060304.so.1
> > 3703e0d000-3703e0e000 rw-p 0000d000 fd:00 1179881                        
> > /lib64/libgcc_s-4.1.0-20060304.so.1
> > 2afc1ca47000-2afc1ca4b000 rw-p 2afc1ca47000 00:00 0
> > 2afc1ca5d000-2afc1ca5e000 rw-p 2afc1ca5d000 00:00 0
> > 2afc1ca5e000-2afc1ca7b000 r-xp 00000000 fd:00 8061031                    
> > /usr/lib64/libg2c.so.0.0.0
> > 2afc1ca7b000-2afc1cb7b000 ---p 0001d000 fd:00 8061031                    
> > /usr/lib64/libg2c.so.0.0.0
> > 2afc1cb7b000-2afc1cb7c000 rw-p 0001d000 fd:00 8061031                    
> > /usr/lib64/libg2c.so.0.0.0
> > 2afc1cb7c000-2afc1cb82000 rw-p 2afc1cb7c000 00:00 0
> > 2afc1cc00000-2afc1cc21000 rw-p 2afc1cc00000 00:00 0
> > 2afc1cc21000-2afc1cd00000 ---p 2afc1cc21000 00:00 0
> > 7fffffe77000-7fffffe8e000 rw-p 7fffffe77000 00:00 0                      
> > [stack]
> > ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0                  
> > [vdso]
> > IOT Trap
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > -----Original Message-----
> > From: Unidata GEMPAK Support [mailto:address@hidden]
> > Sent: Tue 7/3/2007 12:31 PM
> > To: Leonard H. Montenegro
> > Cc: address@hidden
> > Subject: [GEMPAK #DXU-339106]: gempak stopped making images
> >
> >
> > If you kill a program, or one dies unexpectedly, ensure you haven't
> > hung any message queues or left devide drivers and gplt processes running.
> >
> > Run the cleanup script to remove message queues and kill gplt
> > and device processes, eg: "cleanup -c"
> >
> > Steve Chiswell
> > Unidata User Support
> >
> > > Hello,
> > >
> > > Our GEMPAK setup has stopped making images and the process needs to be
> > > killed.
> > >
> > > The last few lines of our log files are below.
> > >
> > > I would appreciate any help in this matter.
> > >
> > > Cordially,
> > >
> > > Leonard
> > >
> > >
> > >
> > > ...
> > >
> > > gdcntr[0x404387]
> > >
> > > gdcntr[0x4fa9c6]
> > >
> > > /lib64/libc.so.6(__libc_start_main+0xf4)[0x36fd91d084]
> > >
> > > gdcntr[0x403759]
> > >
> > > ======= Memory map: ========
> > >
> > > 00400000-00534000 r-xp 00000000 fd:00 3178607
> > > /home/wrf/GEMPAK5.10.1/os/linux64/bin/gdcntr
> > >
> > > 00633000-00639000 rw-p 00133000 fd:00 3178607
> > > /home/wrf/GEMPAK5.10.1/os/linux64/bin/gdcntr
> > >
> > > 00639000-0172b000 rw-p 00639000 00:00 0
> > > [heap]
> > >
> > > 36fd700000-36fd719000 r-xp 00000000 fd:00 1179877
> > > /lib64/ld-2.4.so
> > >
> > > 36fd819000-36fd81a000 r--p 00019000 fd:00 1179877
> > > /lib64/ld-2.4.so
> > >
> > > 36fd81a000-36fd81b000 rw-p 0001a000 fd:00 1179877
> > > /lib64/ld-2.4.so
> > >
> > > 36fd900000-36fda3f000 r-xp 00000000 fd:00 1179878
> > > /lib64/libc-2.4.so
> > >
> > > 36fda3f000-36fdb3f000 ---p 0013f000 fd:00 1179878
> > > /lib64/libc-2.4.so
> > >
> > > 36fdb3f000-36fdb43000 r--p 0013f000 fd:00 1179878
> > > /lib64/libc-2.4.so
> > >
> > > 36fdb43000-36fdb44000 rw-p 00143000 fd:00 1179878
> > > /lib64/libc-2.4.so
> > >
> > > 36fdb44000-36fdb49000 rw-p 36fdb44000 00:00 0
> > >
> > > 36fdc00000-36fdc80000 r-xp 00000000 fd:00 1179880
> > > /lib64/libm-2.4.so
> > >
> > > 36fdc80000-36fdd80000 ---p 00080000 fd:00 1179880
> > > /lib64/libm-2.4.so
> > >
> > > 36fdd80000-36fdd81000 r--p 00080000 fd:00 1179880
> > > /lib64/libm-2.4.so
> > >
> > > 36fdd81000-36fdd82000 rw-p 00081000 fd:00 1179880
> > > /lib64/libm-2.4.so
> > >
> > > 3703d00000-3703d0d000 r-xp 00000000 fd:00 1179881
> > > /lib64/libgcc_s-4.1.0-20060304.so.1
> > >
> > > 3703d0d000-3703e0d000 ---p 0000d000 fd:00 1179881
> > > /lib64/libgcc_s-4.1.0-20060304.so.1
> > >
> > > 3703e0d000-3703e0e000 rw-p 0000d000 fd:00 1179881
> > > /lib64/libgcc_s-4.1.0-20060304.so.1
> > >
> > > 2ba6626c6000-2ba6626ca000 rw-p 2ba6626c6000 00:00 0
> > >
> > > 2ba6626dc000-2ba6626dd000 rw-p 2ba6626dc000 00:00 0
> > >
> > > 2ba6626dd000-2ba6626fa000 r-xp 00000000 fd:00 8061031
> > > /usr/lib64/libg2c.so.0.0.0
> > >
> > > 2ba6626fa000-2ba6627fa000 ---p 0001d000 fd:00 8061031
> > > /usr/lib64/libg2c.so.0.0.0
> > >
> > > 2ba6627fa000-2ba6627fb000 rw-p 0001d000 fd:00 8061031
> > > /usr/lib64/libg2c.so.0.0.0
> > >
> > > 2ba6627fb000-2ba662801000 rw-p 2ba6627fb000 00:00 0
> > >
> > > 2ba662900000-2ba662921000 rw-p 2ba662900000 00:00 0
> > >
> > > 2ba662921000-2ba662a00000 ---p 2ba662921000 00:00 0
> > >
> > > 7fffffc32000-7fffffc48000 rw-p 7fffffc32000 00:00 0
> > > [stack]
> > >
> > > ffffffffff600000-ffffffffffe00000 ---p 00000000 00:00 0
> > > [vdso]
> > >
> > > IOT Trap
> > >
> > > **********************************************************************
> > > NOTICE: This e-mail (and any attachments) may contain PRIVILEGED OR 
> > > CONFIDENTIAL information and is intended only for the use of the specific 
> > > individual(s) to whom it is addressed.  It may contain information that 
> > > is privileged and confidential under state and federal law.  This 
> > > information may be used or disclosed only in accordance with law, and you 
> > > may be subject to penalties under law for improper use or further 
> > > disclosure of the information in this e-mail and its attachments. If you 
> > > have received this e-mail in error, please immediately notify the person 
> > > named above by reply e-mail, and then delete the original e-mail.  Thank 
> > > you.
> > >
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: DXU-339106
> > Department: Support GEMPAK
> > Priority: Normal
> > Status: Closed
> >
> >
> > **********************************************************************
> > NOTICE: This e-mail (and any attachments) may contain PRIVILEGED OR 
> > CONFIDENTIAL information and is intended only for the use of the specific 
> > individual(s) to whom it is addressed.  It may contain information that is 
> > privileged and confidential under state and federal law.  This information 
> > may be used or disclosed only in accordance with law, and you may be 
> > subject to penalties under law for improper use or further disclosure of 
> > the information in this e-mail and its attachments. If you have received 
> > this e-mail in error, please immediately notify the person named above by 
> > reply e-mail, and then delete the original e-mail.  Thank you.
> >
> 
> 
> Ticket Details
> ===================
> Ticket ID: DXU-339106
> Department: Support GEMPAK
> Priority: Normal
> Status: Closed
> 
> 
> **********************************************************************
> NOTICE: This e-mail (and any attachments) may contain PRIVILEGED OR 
> CONFIDENTIAL information and is intended only for the use of the specific 
> individual(s) to whom it is addressed.  It may contain information that is 
> privileged and confidential under state and federal law.  This information 
> may be used or disclosed only in accordance with law, and you may be subject 
> to penalties under law for improper use or further disclosure of the 
> information in this e-mail and its attachments. If you have received this 
> e-mail in error, please immediately notify the person named above by reply 
> e-mail, and then delete the original e-mail.  Thank you.
> 


Ticket Details
===================
Ticket ID: DXU-339106
Department: Support GEMPAK
Priority: Normal
Status: Closed