[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050810:Re: IRaDS & LDM

Hi Marc and A.J.,

Your question was forwarded to me, being a LEAD team member myself.

First, why do you want to decode this data? Are you aware that nexradII is a decompressor? Generally it's beneficial to only decompress this data on demand because the overall compressed volume is very large and the uncompressed volume is huge. Our visualization tools such as the IDV decompress it dynamically, so we generally do not recommend decompressing this as it comes in.

But, to answer your question, when a decoder is spawned depends on the products that come in the data stream. Whether a decoder persists or not and when it terminates depends on how the decoder was coded. Generally we recommend that decoders stay open and read from stdin in order to avoid the overhead of restarting them. The nexradII decoder is slightly different in that it accumulates pieces of a volume scan, decompresses them, and creates a file containing a decompressed volume scan, after which it terminates.

To say more, spawning of a decoder depends on the products that match the regular expression that filters the products sent to the decoder. As long as products come in that match the regular expression and don't cause new substitutions in the template, in this case the "data/pub/decoded/nexradII/raw/\2/\3.raw" string, they will be fed to that same instance of the decoder if it is still running.

So, in this example, when new products come in from a different station or with a different date, this causes different substitutions in the \2 and \3 pieces of the template. Thus, a new instance of the decoder will be spawned with the new values substituted into the template, creating a new argument for this particular invocation. In this data stream the volume scan is sent in a sequence of 100 radial chunks. Since all the products for a particular volume scan have the same station ID and date, this results in a new instance of the decoder for each volume scan from a particular station, and a resulting file that is the complete decompressed volume scan for each instance of the decoder.

Regarding shutting down, this decoder looks for a final product to know when to end the decompression, close the file, and shut down. (Thus, order of arrival is important with this decoder.)

Does this answer your questions?


Unidata Support wrote:
------- Forwarded Message

To: Andrew J Ragusa <address@hidden>
cc: address@hidden,
cc: address@hidden
From: Marcus A Christie <address@hidden>
Subject: Re: IRaDS & LDM
Organization: UCAR/Unidata
Keywords: 200508102225.j7AMP6jo020518

Andrew J Ragusa wrote:

I'm A.J. working with Professor Plale over at IU.  I had a question on the
nexradII data and the LDM decoder.  Mainly, how does LDM launch the
decoder? Is the decoder launched every time a new chunk of a file comes
in, or is it a persistent pipe that starts when the first chunk of file
is received and finished when the last chunk is received??


AJ Ragusa

Just to further elaborate, this is in relation to the LDM pqact action PIPE. The full command, if it helps, is:

NEXRD2 ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9])
PIPE nexradII data/pub/decoded/nexradII/raw/\2/\3.raw


-- *************************************************** Anne Wilson UCAR Unidata Program address@hidden P.O. Box 3000 Boulder, CO 80307 ---------------------------------------------------- Unidata WWW server http://my.unidata.ucar.edu/ ****************************************************

NOTE: All email exchanges with Unidata User Support are recorded in the Unidata inquiry tracking system and then made publicly available through the web. If you do not want to have your interactions made available in this way, you must let us know in each email you send to us.