[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[McIDAS #XDP-858749]: Mcidas Memory problems

Hi Mary Ellen,

> for the second time in a week our McIDAS jobs launched from the crontab have
> begun failing due to a memory problem. The error message is:
> "Cannot make positive UC: could not create 384320-byte shared memory segment"

This is typically an indication that McIDAS sessions (a cron-initiated
script that does McIDAS processing is a session in the same way that an
interactive session is) are not being setup and exited correctly. If
the processing script is not setup correctly (meaning that environment
variables needed by McIDAS are not created correctly) or is interrupted
and so does not shutdown correctly, shared memory segments that have
been created by the user running the script(s) are not returned back
to the OS.  When enough shared memory segments are left, the OS runs out
of memory needed to create those shared memory segments.

> I can log on to the machine as user "mcidas" and successfully run commands
> but not as user "sleet" which owns and runs the scripts through the cron.
> For example, a simple aka.k LIST command produces:
> address@hidden:~/mcidas/data$ aka.k LIST
> aka.k: Cannot make positive UC: could not create 384320-byte shared memory 
> segment

Try this:

- logon as 'sleet' and run:

ipcs -a

I am willing to bet that there are a number of shared memory segments owned by
'sleet'.  These can be cleared using the 'ipcrm' utility; do a 'man ipcrm'
on your machine to see how to run it.

The other thing that will be created along with the shared memory segments
are subdirectories of the .mctmp directory in the HOME directory of the
user.  Along with running 'ipcs -a', also run:

<as 'sleet'>
ls -alt ~sleet/.mctmp

I am willing to bet that there are LOTS of subdirectories listed.  These
can be removed by 'sleet'.

> When this happened the first time, by simply logging onto the machine
> everything was fixed. Literally Randy and did nothing except log on and it
> started working.

This is surprising.

> The symptom is back and a simple log on is not clearing things up this time.

A simple login is not the solution even if it works.

> We rebooted the machine and that fixed things this time.

Rebooting the machine will forceably remove all of the shared memory
segments for all users.  It will not, however, delete the subdirectories
created by McIDAS (the subdirectories of ~user/.mctmp).

> Obviously, we don't
> want to have to reboot this machine every 3-4 days but rather we need to
> determine what is causing the memory overload or how to flush the memory
> nightly.

The "overload", as you call it, is caused by scripts running McIDAS commands
either not being setup correctly, or not terminating correctly.  Again, when
this happens the shared memory segments that are created do not get deleted.
When this happens enough times, there is no shared memory left out of which
new shared memory segments can be created.

The fix will likely be twofold:

- investigate carefully the scripts being running and fix whatever problems
  that they have

- make sure that your OS is configured to have enough shared memory

  The Unidata McIDAS User Guide has a section on setting the shared memory
  in various OSes:

Unidata HomePage

  Software -> McIDAS

    Documentation & Training

      McIDAS user's guide

Installing and Configuring McIDAS-X

  Installing McIDAS-X on Unix or Mac OS X Workstations

    Preparing the Workstation

      Allocating Sufficient Shared Memory -> Shared Memory Configuration

> The issue is critical because of our data download schedules and
> realtime processing are being interrupted.

I understand.

> Any thoughts would be greatly appreciated.

Please review what I wrote above and let me know if you have any questions.

> Thanks so much!

No worries.


Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
Unidata HomePage                       http://www.unidata.ucar.edu

Ticket Details
Ticket ID: XDP-858749
Department: Support McIDAS
Priority: Normal
Status: Closed

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.