[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

20041217: Question about LDM data format

>From: "John R. Jackson" <address@hidden>
>Organization: Purdue
>Keywords: 200412172351.iBHNpUlI025223

Hi JJ-

>I have what I'm sure will be an incredibly naive question and hope you'll
>be able to at least point me in the right direction.
>Now that we're providing the Level-II data to some folks, one of them
>is actually being so silly as to want to look at it.  Unfortunately, I'm
>just the system admin of the delivery mechanism and haven't had training
>(or time) on the data itself, so I don't know how to help them decode
>what they are getting.

That's what happens when you have users.  Always wanting more. ;-)

>According to them, they are used to something called ARCHIVE2 format but
>what I'm giving them is AR2V0001.  There is also some mention of "real
>time" format vs "differ time" format (although I may have misunderstood
>that -- there is a bit of a language barrier).

The AR2V0001 is a new format that adds in some (necessary information) to
the 24 byte header at that beginning of the file.  It adds in an actual
time and the 4 letter ID of the radar location.  In a regular ARCHIVE2
file, there is not information on where the data are located.

>I am providing what I call "GEMPAK" files to them, which are the result
>of cat-ing the radial slices of a scan together until the end marker ('E')
>product is received.  This is the equivalent of your dccraft_move script.

What is sent over the IDD feed is a series of packets that contain
either the uncompressed header, or bzip2 compressed packets of a
series of radials (100 or so).  Each packet has an uncompressed integer
specifying the number of compressed bytes to follow.
The pqact entries you are using will write the header and compressed 
radials to a temporary file and when it gets the E file, it will know
it's done and move the temp file to its permanent position.  This is
done so you won't get incomplete displays if auto-update is on in

>Dr Huber found that the GEMPAK dcnexr2 utility can read the files.
>I have no idea what it's doing but I was able to run it on a sample file
>I had sent them and it produced a log file implying it was able to parse
>the input (and it generated some kind of output file).

The dcnexr2 program will take the compressed files and uncompress them
into the AR2V0001 format which is essentially the ARCHIVE2 format with
the extra metadata in the header.  If they have programs that read
ARCHIVE2 format, they should just be able to modify the code to accept
the AR2V0001 format as well.  

>I was also able to point IDV at the files, tell it they were Level-II
>format and it put up some kind of picture (although I don't know what I
>was looking at).  I figured since it didn't drop core :-) it must have
>been reasonably happy with the data.

The IDV was modified to decode the bzip2 files just as GEMPAK does.
Many sites just want to save disk space and leave them in compressed

>I passed along the above information but it didn't seem to give them
>the answer they were looking for.

If they want to use the AR2V0001 format with a program that reads
ARCHIVE2 format, they will at least have to uncompress them using 
dcnexr2.  If the program is looking for a file that starts with
ARCHIVE2, then it will have to be modified to also understand
files that begin with AR2V0001 as well.  The internal format is
essentially the same (except for the header).

>So ...
>Does any of this make even the slightest amount of sense?

Yes.  Does my answer?

>Is this the kind of question Unidata Support helps with?  Or some
>other group you know of that you could direct me to?

Sure, that's what we are here for.

>Is the answer so blindingly obvious I should have been able to just
>point them at some document?

Relevant docs are:

http://www.ncdc.noaa.gov/oa/radar/leveliidoc.html (level II format)

http://www.roc.noaa.gov/NWS_Level_2/Web_FAQs.pdf  (info on Level II feed)

>What are ARCHIVE2 and ARV0001 formats and where are they defined (or
>do I care)?

They are essentially the same.  However, as you are filing them,
the AR2V0001 files are compressed so they need to be uncompressed
unless you are trying to read them in GEMPAK or IDV (or some other
program that understands the compressed format).  Plus, if you
uncompress them, you'll need lots of disk space.

>Do I need to restart my college days and become a Geo Scientist in my
>spare time?

Never hurts to learn more.

Don Murray
NOTE: All email exchanges with Unidata User Support are recorded in the
Unidata inquiry tracking system and then made publicly available
through the web.  If you do not want to have your interactions made
available in this way, you must let us know in each email you send to us.

>From address@hidden  Tue Dec 21 21:55:40 2004

>The AR2V0001 is a new format ...

Thanks for the information.  I passed it along to the user and we'll
see if it helps.  I also read enough of it to get the general idea myself.