[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20041026: Receive, compress, and send

>From: "Keen.Jeremy" <address@hidden>
>Organization: UCAR/Unidata
>Keywords: 200410261316.i9QDGcvV018871

>I was wondering if there is a way in which to receive data from an upstream LD
> M to our main downstream LDM pull what we want from that stream with our pqac
> t then compress the data and send the compressed data to a downstream LDM fro
> m our main LDM?  Will greatly appreciate any help you can give on this.  
> 
>Thanks
>Jeremy
>
--

Jeremy,

You can "PIPE -close" the incoming product to a script which compresses
the data product (compress, gzip, bzip2), and then runs pqinsert to
insert the created compressed product into your queue. Your downstream
can receive the products you place in your queue by requesting the appropriate
feedtype/pattern.

An example of a csh script could look like:

#!/bin/csh -f
# Copyright (c)2003 UCAR/Unidata
#
# Permission to use, copy, modify, and distribute this software and its
# documentation for any purpose without fee is hereby granted, provided
# that the above copyright notice appear in all copies, that both that
# copyright notice and this permission notice appear in supporting
# documentation, and that the name of UCAR/Unidata not be used in
# advertising or publicity pertaining to distribution of the software
# without specific, written prior permission.  UCAR makes no
# representations about the suitability of this software for any purpose.
# It is provided "as is" without express or implied warranty.  It is
# provided with no support and without obligation on the part of UCAR or
# Unidata, to assist in its use, correction, modification, or enhancement.
#
# Steve Chiswell        10/26/2004
#
echo "$argv" | logger -t "$0 [$$]" -p local0.notice

cat | gzip > data/${argv}
pqinsert -f EXP -p ${argv} data/${argv}

exit(0)



The above script assumes that gzip and pqinsert are in the path of the running
pqact process. Your pqact.conf action would look like
ANY     somepattern
        PIPE    -close  util/compress_script.csh prodname.gz


The invocation will create an ldmd.log line for your script with the time and 
prodname.gz
so you can track your processing. The $argv will be prodname.gz.
The -p flag to pqinsert lets you specify the product name inserted into the 
queue,
and if ommitted the product name will be the compete file path name.
Depending on your needs, you can remove the data product after queue insertion,
or scour it later as needed.


Steve Chiswell

NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.