[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: 20050810:Re: IRaDS & LDM

Hi Marc and A.J.,

Your question was forwarded to me, being a LEAD team member myself.
First, why do you want to decode this data? Are you aware that nexradII is a decompressor? Generally it's beneficial to only decompress this data on demand because the overall compressed volume is very large and the uncompressed volume is huge. Our visualization tools such as the IDV decompress it dynamically, so we generally do not recommend decompressing this as it comes in.
But, to answer your question, when a decoder is spawned depends on the products that come in the data stream. Whether a decoder persists or not and when it terminates depends on how the decoder was coded. Generally we recommend that decoders stay open and read from stdin in order to avoid the overhead of restarting them. The nexradII decoder is slightly different in that it accumulates pieces of a volume scan, decompresses them, and creates a file containing a decompressed volume scan, after which it terminates.
To say more, spawning of a decoder depends on the products that match the regular expression that filters the products sent to the decoder. As long as products come in that match the regular expression and don't cause new substitutions in the template, in this case the "data/pub/decoded/nexradII/raw/\2/\3.raw" string, they will be fed to that same instance of the decoder if it is still running.
So, in this example, when new products come in from a different station or with a different date, this causes different substitutions in the \2 and \3 pieces of the template. Thus, a new instance of the decoder will be spawned with the new values substituted into the template, creating a new argument for this particular invocation. In this data stream the volume scan is sent in a sequence of 100 radial chunks. Since all the products for a particular volume scan have the same station ID and date, this results in a new instance of the decoder for each volume scan from a particular station, and a resulting file that is the complete decompressed volume scan for each instance of the decoder.
Regarding shutting down, this decoder looks for a final product to know when to end the decompression, close the file, and shut down. (Thus, order of arrival is important with this decoder.) 

Does this answer your questions?

Anne


Unidata Support wrote:

------- Forwarded Message



To: Andrew J Ragusa <address@hidden>
cc: address@hidden,
cc: address@hidden
From: Marcus A Christie <address@hidden>
Subject: Re: IRaDS & LDM
Organization: UCAR/Unidata
Keywords: 200508102225.j7AMP6jo020518



Andrew J Ragusa wrote:


I'm A.J. working with Professor Plale over at IU.  I had a question on the
nexradII data and the LDM decoder.  Mainly, how does LDM launch the
decoder? Is the decoder launched every time a new chunk of a file comes
in, or is it a persistent pipe that starts when the first chunk of file
is received and finished when the last chunk is received??

Thanks

AJ Ragusa
address@hidden
630-697-2470
Just to further elaborate, this is in relation to the LDM pqact action PIPE. The full command, if it helps, is:
NEXRD2 ^L2-([^/]*)/(....)/([0-9][0-9][0-9][0-9][0-1][0-9][0-3][0-9][0-2][0-9][0-5][0-9][0-9][0-9]) 
         PIPE    nexradII data/pub/decoded/nexradII/raw/\2/\3.raw


Marc



--
***************************************************
Anne Wilson                     UCAR Unidata Program            
address@hidden                 P.O. Box 3000
                                  Boulder, CO  80307
----------------------------------------------------
Unidata WWW server       http://my.unidata.ucar.edu/
****************************************************