[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Datastream #YEX-601459]: Reading NEXRAD Level II Data

> Hello!

Hello Tristan, 

> 
> I am stuck. I've downloaded data from NCEI (KFFC20170513_130002_V06.ar2v)
> and am trying to write a simple algorithm to decode the data (I am
> well-aware of the many tools out there, I want to do this myself for my
> own work).
> 
OK

> I've read documentation from:
> 
> ftp://ftp.ncdc.noaa.gov/pub/data/radar/NEXRADFormatChangeAnnouncement-20160317.pdf
> https://www.roc.noaa.gov/wsr88d/BuildInfo/Files.aspx
> 
> file "ICD FOR RDA/RPG - Build RDA 17.0/RPG 17.0 (PDF)" from:
> https://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/RDA_RPG_2620002P.pdf
> 

Yes, we would point you to the ICD as well for information.

> I've read code from the RSL software and the Java decoder (just to
> see how some of it works and the byte order of the NEXRAD-II data):
> 

Yes, we have incorporated some of the RSL software into our applications.

> https://trmm-fc.gsfc.nasa.gov/trmm_gv/software/rsl/
> 
> Specifically: wsr88d.c, wrs88d_decode_ar2v.c, and wsr88d_m31.c (of which,
> I know these are not your programs, I can follow the code for some time,
> and then I get lost because I'm not that hardcore of software developer
> yet).
> 
> The java decoder:
> 
> https://github.com/gss2002/nexradtest/blob/master/src/main/java/org/senia/nexrad/l2/ScanTest.java
> 
> I've read the bzip2 library documentation
> 
> http://www.bzip.org/1.0.5/bzip2-manual-1.0.5.html
> 
> and have tried BZ2_bzBuffToBuffDecompress and BZ2_bzDecompress functions.
> 
> So now you understand I have extensively researched this and am stuck...
> FYI, portions of this program are borrowed from RSL. They are similar to
> this thread:
> 
> https://www.unidata.ucar.edu/support/help/MailArchives/datastream/msg01661.html
> 
> My problem:
> 
> I can read the 28-byte header in the .ar2v file:
> 
> /* Check for AR2V... */
> fread(version, 1, 4, f);
> printf("HEADER **%s**\n", version);
> fseek(f, 16, SEEK_CUR);
> char stid[4];
> fread(stid, 1, 4, f);
> printf("STIN: ***%s***\n", stid);
> /*and get the remaining 4 bytes for the byte length of the compressed data:*/
> fread(&comp_length, 4, 1, f);
> for(i=0;i<4;i++)
> length = ( length << 8 ) + (unsigned char)comp_length[i];
> if(length<0) length*=-1;
> printf("Length of compressed block: %d\n", length);
> 
> Then I run a while loop (similar to RSL wsr88d_decode_ar2v.c in RSL library)
> with the main decompression portion being:
> 
> while ( bzerror == BZ_OK ) {
> size_t nbytes = 0;
> nbytes = fread(block, 1, length, f);
> printf("Bytes read: %lu\n", nbytes);
> if(nbytes != length) {
> printf("Bytes do not equal each other.\n"
> "Expected %d, got %lu\n", length, nbytes);
> return -3;
> }
> 
> tryagain:
> olength = osize;
> 
> bzerror = BZ2_bzBuffToBuffDecompress(oblock, &olength, block, length, 0, 0);
> printf("ERRORCODE: %d\n", bzerror);
> 
> if(bzerror) {
> if (bzerror == BZ_OUTBUFF_FULL) {
> osize += 262144;
> if ((oblock=(char*) realloc(oblock, osize)) == NULL) {
> fprintf(stderr, "Cannot allocate output buffer\n");
> exit(1);
> }
> goto tryagain;
> } else {
> printf("Error: %d\n", bzerror);
> return bzerror;
> }
> }
> fwrite(oblock, 1, olength, outbin);
> return bzerror;
> }
> 
> I only read the first chunk of compressed data for simplicity. I print
> the buffer which it is stored in (oblock) via fwrite() to a file called
> out.bin (real original).
> 
> I run this, and all my return error codes are 0 from bzlib.h (BZ_OK). A
> file is created with data in it. This is [hopefully] the decompressed
> data from the first block of the .ar2v file. Is this data binary? I'm
> assuming it is?
> 
> Now here's the part that is stumping me...  To parse the Message 31 data
> (which I'm assuming it is?) I've been looking at Table XVII from
> 
> https://www.roc.noaa.gov/wsr88d/PublicDocs/ICDs/RDA_RPG_2620002P.pdf
> 
> According to this, bytes 0-3 should be the station id...  My parsing
> program is simply written which reads from out.bin created above (the
> output first block of .ar2v decompressed data):
> 
> char buf[4];
> [file io stuff]
> fread(buf, 1, 4, fp);
> printf("STATION ID: %s\n", buf);
> 
> According to that documentation, the Station ID should be printed out,
> instead it prints blank. Nothing.
> 
It is possible that these fields are blank, we use look up tables and the file 
name to define station ID.


> Is what I'm trying to do feasible? Essentially, printing all the
> information in the compressed block to the screen. I want to ***see***
> what's in the file.


 I would think so.

> 
> I have been working on this for a few days now straight, and am going
> a little bonkers. My slight concern is that I'm reading the wrong
> documentation and/or expecting something that isn't actually there. For
> instance, Table XVII in the above pdf says byte location 16 should be
> the "Compression Indicator". What the heck is that information doing in
> the compressed data? Having to decompress the data without knowing what
> compression method is used to find out what compression method was used
> is backwards and has my eyes rolling back into my head.
> 
> I feel like I'm on to something having gotten past the "header" to
> find how many bytes are stored in the first chunk, but I can't read the
> compressed data even though I'm pretty darn sure I've decompressed it
> to an output file.
> 
> Any and all help is greatly appreciated. Thank you so much for your
> time. I know it's a lot.
> 
> -Tristan
> 


We do not have resources to do a code review for you, but we are happy to test 
your file to make sure we can visualize it
with our tools, and give a check on the files fidelity. If you would like us to 
do that, please send the file along.




> My full program is below:
> #include <stdlib.h>
> #include <stdio.h>
> #include <string.h>
> #include <bzlib.h>
> #include <unistd.h>
> 
> #define maxsize  100
> 
> int parsedata (char *infile)
> {
> 
> FILE*   f;
> BZFILE* b;
> int     nBuf;
> char    buf[maxsize];
> char    version[4];
> char    header[4];
> char    comp_length[4];
> int     bzerror;
> int     nWritten;
> char *outbuf;
> outbuf = malloc(maxsize);
> 
> /* BORROWING FROM RSL... */
> int length=0;
> unsigned isize = 8192, osize&2144, olength;
> char *block = (char *)malloc(8192), *oblock = (char *)malloc(262144);
> 
> f = fopen ( infile, "r" );
> if ( !f ) {
> printf("ERROR. Goodbye\n");
> exit(2);
> }
> 
> /* Later iterations: place a check for "message 31" format (V>1)
> * e.g.: AR2V0006
> */
> /* Check for AR2V... */
> fread(version, 1, 4, f);
> printf("HEADER **%s**\n", version);
> fseek(f, 16, SEEK_CUR);
> char stid[4];
> fread(stid, 1, 4, f);
> printf("STID: ***%s***\n", stid);
> 
> /* Up to this point, 24 bytes have been read/
> * Byte location 29 is where bzip 2 begins.
> * The 4 bytes remaining in the "header" are the number
> *   of bytes that are compressed in the bzip2 compressed data.
> */
> 
> int i;
> fread(&comp_length, 4, 1, f);
> for(i=0;i<4;i++)
> length = ( length << 8 ) + (unsigned char)comp_length[i];
> if(length<0) length*=-1;
> printf("Length of compressed block: %d\n", length);
> 
> FILE *outbin;
> if((outbin=fopen("out.bin", "wb")) == NULL) {
> printf("Cannot open outbinary.\n");
> return(-2);
> }
> 
> bzerror = BZ_OK;
> int nnn=0;
> int bytesread;
> 
> while ( bzerror == BZ_OK ) {
> 
> size_t nbytes = 0;
> nbytes = fread(block, 1, length, f);
> printf("Bytes read: %lu\n", nbytes);
> if(nbytes != length) {
> printf("Bytes do not equal each other.\n"
> "Expected %d, got %lu\n", length, nbytes);
> return -3;
> }
> 
> tryagain:
> olength = osize;
> 
> bzerror = BZ2_bzBuffToBuffDecompress(oblock, &olength, block, length, 0, 0);
> printf("ERRORCODE: %d\n", bzerror);
> 
> if(bzerror) {
> if (bzerror == BZ_OUTBUFF_FULL) {
> osize += 262144;
> if ((oblock=(char*) realloc(oblock, osize)) == NULL) {
> fprintf(stderr, "Cannot allocate output buffer\n");
> exit(1);
> }
> goto tryagain;
> } else {
> printf("Error: %d\n", bzerror);
> return bzerror;
> }
> }
> fwrite(oblock, 1, olength, outbin);
> return bzerror;
> }
> 
> fclose(outbin);
> return 0;
> }
> 
> int main(int argc, char* argv[])
> {
> if(argc!=2) {
> printf("Need infile in arg 1\n");
> exit(2);
> }
> char *infile;
> infile = argv[1];
> printf("Infile: %s\n",infile);
> int ret = parsedata(infile);
> printf("Main concluded with: %d\n", ret);
> return 0;
> }
> 
> 
> 

Cheers, 

Jeff
Jeff Weber
Unidata User Support
http://www.unidata.ucar.edu

Ticket Details
===================
Ticket ID: YEX-601459
Department: Support Datastream
Priority: High
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.