[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[LDM #DPA-233086]: Automatic insertion of products into queue

Benjamin,

> One of the faculty members here generates daily WRF model output on a
> research cluster.  Currently the data is primarily distributed as images on
> a web server, but we'd like to send it out via our LDM server in order to
> provide easier access.  The first step is to figure out the best way to
> automate the process.  From what I've been able to gather from the LDM
> manpages, it seems pqinsert is the appropriate command.  As a test, I was
> able to send data using the following command
> 
> pqinsert -v -l - -f EXP /project/wxp/a/wxp/www/html/wrfdata/dailyrun/*
> 
> While that successfully got data from the server to a test recipient I set
> up, it's not clear to me how it should be automated.  The same command run
> every few minutes or so from cron would have to re-process every file and
> error on the ones that are already in the queue (and potentially re-send
> files that have expired out of the queue).

We have several sites that use pqinsert(1) to distribute data.  The trick
is to run pqinsert(1) from a script that removes the file if pqinsert(1)
terminates successfully.  That way you don't have to worry about duplicating
files.

> Another option would be to give pqinsert as an -exec argument to find, but
> that seems...like something I would do. :-)
> 
> It could be that I've missed something, but it seems like most of the
> documentation is (understandably) written toward receiving data, not
> originating.  Any advice you could provide would be greatly appreciated.
> 
> 
> Thanks,
> BC
> 
> --
> Ben Cotton
> Unix Systems Administrator
> Department of Earth and
> Atmospheric Sciences
> Purdue University
> http://www.purdue.edu/eas/resources/it/
> 
> CIVL 4212
> O: 765-49-61546
> C: 765-404-7986


Regards,
Steve Emmerson

Ticket Details
===================
Ticket ID: DPA-233086
Department: Support LDM
Priority: Normal
Status: Closed