[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #QST-811359]: Using pqinsert as daemon



Hi Justin,

re:
> We're in the process of setting up a new LDM dissemination system based on
> the call we had with you a few months ago. We're still a bit away from even
> being able to test, but we're starting to look into how we'll configure
> this system.

Thinking through the process now is a very good idea.

re:
> One issue we need to resolve is inserting new products into the LDM queue
> using pqinsert.
> 
> Here is what we are currently thinking...
> 
> 1. External processes write files to a shared directory
> 2. A have a daemon scanning that directory and inserting each product into
>    the queue via pqinsert
> 3. Once a file is inserted into the LDM queue is is removed from the
>    directory
> 4. Sleep for a second or two
> 5. Scan the directory again for new products

This all sounds reasonable.  There will be some "dotting of Is and crossing
of Ts" as the procedure is fleshed-out, of course.

re:
> Now that is how we've handled inserting files into LDM on our supercomputer
> and there we use pqing. We want to transition to using pqinsert on this new
> system, do you have any thoughts on our plan?

You will want to instrument your "daemon" process to:

- provide full logging capabilities

- send a signal to the LDM process leader to inform the running LDM
  processes that there is/are new products in the queue

  Without doing this, the sending of the data will wait until the
  LDM server (ldmd) process(es) wake up after a 30 second sleep
  and scan the queue for new products to process.  The effect of
  not sending the signal to the LDM process leader would be latencies
  that vary between 0 and 30 seconds.  This may or may not be a huge
  concern, but it is something worth paying attention to.

  If this does not make sense, you may want to consult with Carissa K
  as she should remember this from the LDM training she went through
  here at the UPC offices in Boulder.

re:
> Do you have or know of any
> daemons that exist now that do this type of scanning and inserting?

Not off of the top of my head, but we know that there are numerous
folks doing this sort of thing in the community.  You could ask them
for their input by posting an email to the address@hidden
email list that we maintain on behalf of the community.

NB: before you can post to any of the email lists we maintain, you
must be subscribed to that/those list(s).  You can (un)subscribe
to any list we maintain online at:

Unidata HomePage
http://www.unidata.ucar.edu

  Support -> Mailing Lists
  http://www.unidata.ucar.edu/support/index.html#mailinglists

Click on the 'show mailing list actions' link for each mailing
list you want to (un)subscribe to and choose the appropriate
option.

re:
> Thanks again for your help,

No worries.

Cheers,

Tom
--
****************************************************************************
Unidata User Support                                    UCAR Unidata Program
(303) 497-8642                                                 P.O. Box 3000
address@hidden                                   Boulder, CO 80307
----------------------------------------------------------------------------
Unidata HomePage                       http://www.unidata.ucar.edu
****************************************************************************


Ticket Details
===================
Ticket ID: QST-811359
Department: Support LDM
Priority: Normal
Status: Closed