Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts

  • To: "ldm-users@xxxxxxxxxxxxxxxx" <ldm-users@xxxxxxxxxxxxxxxx>
  • Subject: Re: [ldm-users] 20200423: Re: Efficiency of splitting pqacts
  • From: "Herzmann, Daryl E [AGRON]" <akrherz@xxxxxxxxxxx>
  • Date: Thu, 23 Apr 2020 19:36:50 +0000
  • Arc-authentication-results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=iastate.edu; dmarc=pass action=none header.from=iastate.edu; dkim=pass header.d=iastate.edu; arc=none
  • Arc-message-signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=zzE8OcEBAwRnOklyO/nMm0W/q7O2RmNPepMemJiMLG0=; b=G6VFd7/Eq0FpguOmUERvwb1mdKpJnItdyGZaL3X1LRoT5mNSuVxlaInmEpBeyxfb7Me/bUPhmur9psWA/ys0LGT8Kih2EqBqQrmcn5Yi0iz7Y24nZEh7yQBLBPMpLMIxLtoaHECOrq+gcd23dcJLXBlj3q0gWlKYG5CGkP2Mt2I9mxU8hLjiWCEEfEcRGNtft+dUJLwLQFmm4y2HsXC2ccTgyxXc1pTdQsD6f1I/oK63ljYwLDirh85Dhq3FZokgeJ+HASR/4HDOoeuVwdHtfzPaau992CWL/d2X9hVQ2kiShdQ49gM6IOOb2795GI2R/Y4l2VqRB0eBHINbIrbx+w==
  • Arc-seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=YGgj3PFQWaIteOjUT5SaMeh7h+PsfqVQQnmssb7C5t6HZOLa0LOdBfuBqiarvM/z1SFqiCnhi5hXqdY5VN26eUZqJPfxhRVupxNo6aRjpIfSE2YIm0NtGRkUO3Bd5xOZvmPvWjMtttm07Y342vBODP3El+pGgEMuDDRksosTvgqdm5AbmWCS9QhysAwmPWdPOVJGhNCBGU1v5kjEFLNbFFtQ4knFL/kswrDkZ6wRtKfGuftz3sIc0TtQHczT965wD8VqvbUn1/LCBQWTxZ1fCRogULDVIR1bYv4yYnoXiwqOgr+4pIir5PboUx7fpTQHzVZbNkHc3LMF91kFaVBdCw==
  • Authentication-results: spf=none (sender IP is ) smtp.mailfrom=akrherz@xxxxxxxxxxx;
Well Howdy,

I am sure Unidata will correct my ignorance / incorrect details, but my 
understanding is that an individual pqact process can only do 32 "things" at 
one time, or there's 32 slots available for work.

Now, the above depends on the action.  If you run `PIPE -close`, the slot can 
be used for another product even with the PIPEd process still running...  This 
type of action can lead LDM to DOSing the server it is on as it will fire off 
as many PIPE'd processes that it can.  You old timers, like me, will recall the 
lock file fun Chiz wrote into the GIF generation script of NIDS data for this 
reason.

If you are doing just FILE actions without a `-close`, there is some benefit to 
spreading out the pqact.conf file into multiple files to keep each pqact 
roughly touching 32 files each.  For example with level2 data, dividing the 
radars into chunks like so:

exec    "pqact -p BZIP2/K[A-D] -f CRAFT /local/ldm/etc/pqact-craft.conf"
exec    "pqact -p BZIP2/K[E-H] -f CRAFT /local/ldm/etc/pqact-craft2.conf"
exec    "pqact -p BZIP2/K[I-K] -f CRAFT /local/ldm/etc/pqact-craft3.conf"
exec    "pqact -p BZIP2/K[L-O] -f CRAFT /local/ldm/etc/pqact-craft4.conf"
exec    "pqact -p BZIP2/K[P-R] -f CRAFT /local/ldm/etc/pqact-craft5.conf"
exec    "pqact -p BZIP2/K[S-Z] -f CRAFT /local/ldm/etc/pqact-craft6.conf"
exec    "pqact -p BZIP2/[A-J] -f CRAFT /local/ldm/etc/pqact-craft7.conf"
exec    "pqact -p BZIP2/[L-Z] -f CRAFT /local/ldm/etc/pqact-craft8.conf"

Behold, another caveat here.  While with the above, each pqact process has its 
own uniquely named file, this file can be the same file on the filesystem and 
managed with sym links.  They need to be unique to the pqact process so that 
pqact can write its `.state` file to a unique location.

You should consider the processes being run, how long their lifetime is, and 
your server's capacity.   If you have a bunch of long running GEMPAK decoders 
that totals something less than 32 total, then just keep them in one file but 
perhaps isolate that pqact process to just those tasks.

So hold tight until Unidata corrects my above as FUD :)

daryl

--
/**
 * daryl herzmann
 * Systems Analyst III -- Iowa Environmental Mesonet
 * https://mesonet.agron.iastate.edu
 */

________________________________________
From: ldm-users <ldm-users-bounces@xxxxxxxxxxxxxxxx> on behalf of Tom Yoksas 
<yoksas@xxxxxxxx>
Sent: Thursday, April 23, 2020 2:15 PM
To: ldm-users@xxxxxxxxxxxxxxxx
Subject: [ldm-users] 20200423: Re:  Efficiency of splitting pqacts

Hi Mike,

On 4/23/20 12:39 PM, Mike Zuranski wrote:
> I'm wondering if there is a difference in speed/efficiency of the LDM,
> or in system resource allocation, between grouping all my pqact
> statements in one file vs. splitting them up into different pqact
> files.

Since all actions in an LDM pattern-action file are processed
sequentially, there is a benefit to distributing actions in multiple
pattern-action files that are each processed by a separate 'pqact'
instance.

re:
> Does LDM do anything differently or is it a wash either way?

No, each 'pqact' instance will work through the list of actions in
the pattern-action file that it works in sequence.  So, if one has
a monolithic pattern-action file with, say 10K actions, it will take
significantly longer than having 10 'pqact' instances operating
on pattern-action files that each have 100 actions.

re:
> I vaguely remember this coming up at one point but I couldn't find any
> documentation or old email threads about it.  I'm mostly just asking out
> of curiosity, I don't have a specific problem that I'm trying to solve
> or anything.  But if I were to redo my pqact organization I'm wondering
> if there is a preferred methodology.

The best rule of thumb is to have multiple 'pqact' instances operating
on multiple pattern-action files when the list of actions to be
performed is large, or when some of the actions are slow.  There is no
"best practice" for, say, having only N actions in a pattern-action
file since the speed that the actions will be performed is a function
of how fast/slow each action is.  Sites invariably will need to do
their own tuning to find the right balance of speed and use of
resources (more 'pqact' instances will, of course, use more resources
like CPU, RAM, etc.).

Cheers,

Tom
--
+----------------------------------------------------------------------+
* Tom Yoksas                                      UCAR Unidata Program *
* (303) 497-8642 (last resort)                           P.O. Box 3000 *
* yoksas@xxxxxxxx                                    Boulder, CO 80307 *
* Unidata WWW Service                     http://www.unidata.ucar.edu/ *
+----------------------------------------------------------------------+

_______________________________________________
NOTE: All exchanges posted to Unidata maintained email lists are
recorded in the Unidata inquiry tracking system and made publicly
available through the web.  Users who post to any of the lists we
maintain are reminded to remove any personal information that they
do not want to be made public.


ldm-users mailing list
ldm-users@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit: 
https://www.unidata.ucar.edu/mailing_lists/


  • 2020 messages navigation, sorted by:
    1. Thread
    2. Subject
    3. Author
    4. Date
    5. ↑ Table Of Contents
  • Search the ldm-users archives: