[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)



Pelin,

attached is Grib1ExtractRawData.java class. Put it in the grib1 dir and compile. it will extract the first record if not parm # is given or all the records of a certain parm, ie

% java -Xmx256m ucar/grib/grib1/Grib1ExtractRawData RUC.grib1 > RUC1rec.grib1

or

% java -Xmx256m ucar/grib/grib1/Grib1ExtractRawData RUC.grib1 7 >
RUCparm7.grib1

can you create a file either 1 record or one with a reasonable # of parms so i can use it as a test case.

RObb...





On Wed, 10 Sep 2008, Bali, Pelin wrote:

Robb,

I can attach the original ecmwf grib file if you want me to do so. It is
about 35MB... When I zip it, it didn't make much difference. Please let
me know how you want to proceed.

If I read the dx and dy values by using readShort method in the
Grib1GridDefinitionSection.java this is how grid definition section
looks like in the  grib index file:

GDSkey = 1015536804
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 720
Ny = 171
La1 = 65.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -20.0
Lo2 = 179.5
Dx = -3.071
Dy = -3.071
ScanningMode = 0
--------------------------------------------------------------------
GDSkey = 3838746175
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 720
Ny = 171
La1 = 65.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -20.0
Lo2 = 179.5
Dx = -3.071
Dy = -3.071
ScanningMode = 0
--------------------------------------------------------------------
GDSkey = 1388487965
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 360
Ny = 181
La1 = 90.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -90.0
Lo2 = 179.0
Dx = -6.141
Dy = -6.141
ScanningMode = 0

If I read the same variables by using readInt2 method in the
Grib1GridDefinitionSection.java, I got the following values:

GDSkey = 1015536804
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 720
Ny = 171
La1 = 65.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -20.0
Lo2 = 179.5
Dx = 0.5
Dy = 0.5
ScanningMode = 0
--------------------------------------------------------------------
GDSkey = 3838746175
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 720
Ny = 171
La1 = 65.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -20.0
Lo2 = 179.5
Dx = 0.5
Dy = 0.5
ScanningMode = 0
--------------------------------------------------------------------
GDSkey = 1388487965
grid_type = 0
grid_name = Latitude/Longitude Grid
grid_shape_code = 0
grid_shape = spherical
grid_radius_spherical_earth = 6367.47
Nx = 360
Ny = 181
La1 = 90.0
Lo1 = 180.0
ResCompFlag = 128
Winds = Relative
La2 = -90.0
Lo2 = 179.0
Dx = 1.0
Dy = 1.0
ScanningMode = 0


Sincerely,

Pelin Bali

-----Original Message-----
From: Robb Kambic [mailto:address@hidden]
Sent: Tuesday, September 09, 2008 5:25 PM
To: Bali, Pelin
Cc: decoders
Subject: RE: Fw: decoders] Grib1 Indexer and GDS info (fwd)

On Thu, 28 Aug 2008, Bali, Pelin wrote:

Robb,

I've been registered to address@hidden and I was sending my
questions to 'address@hidden' all the time. I don't know
how
it is being handled from there and how it falls into your spam. I
started to think that you were ignoring my messages. <:)

After sending this message, I made a little change in my code to read
Grib1GridDefinitionSection.params field to get HashMap of the
parameters. This helped me to get all the information I needed.
However,
in the grib1 index file, the Dx and Dy fields of the grid definition
section was still displaying wrong numbers. My colleague and I started
looking into the Grib1GridDefinitionSection.java. In the section where
the information obtained from octets 24-25 (Dx) and octets 26-27(Dy or
np), they were being read as "short". We changed them to read by using
GribNumbers.int2 method (please see the following code snippet). It
started spitting out the correct numbers.

Sincerely,

Pelin Bali.



Pelin,

Here's my findings so far, the only time readSHort returns a wrong
value
is the missing value scenario. ie when byte 1 and 2 = 255. readShort
returns -0.0010 and GribNumbers.int2 return -9999. Is this what you are
seeing? If not could you make available a file that i can test.

thanks,
robb...






                case 202 :

             // octets 18-20 (La2 - latitude of last grid point)
             lat2 = GribNumbers.int3(raf) / 1000.0;

             // octets 21-23 (Lo2 - longitude of last grid point)
             lon2 = GribNumbers.int3(raf) / 1000.0;

             // octets 24-25 (Dx - Longitudinal Direction Increment )
             //------------------------------------------------------
             // This is where we made the change.
             //-------------------------------------------------------
             //dx = raf.readShort() / 1000.0;
             dx = GribNumbers.int2(raf) / 1000.0;

             // octets 26-27 (Dy - Latitudinal Direction Increment )
             //               Np - parallels between a pole and the
equator
             if (type == 4) {
               //  np = raf.readShort();
             np = GribNumbers.int2(raf);
             } else {
               //  dy = raf.readShort() / 1000.0;
             dy = GribNumbers.int2(raf) / 1000.0;
             }

//-----------------------------------------------------------
             // End of our changes.
             //
----------------------------------------------------------
             // octet 28 (Scanning mode)  See Table 8
             scan = raf.read();
             // octet 29-32 reserved
             reserved = raf.readInt();
             if (P_VorL != 255) {
                 if( NV == 0 || NV == 255 ) {
                     getPL(raf);
                 } else {
                     getPV(NV, raf);
                 }
             }
             break;  // end Latitude/Longitude grids



-----Original Message-----
From: Robb Kambic [mailto:address@hidden]
Sent: Wednesday, August 27, 2008 3:20 PM
To: Bali, Pelin
Cc: decoders
Subject: Re: Fw: decoders] Grib1 Indexer and GDS info (fwd)

Bali,

your support msg has been sitting in my spam folder for months now, i
just
clean it out so that;s the reason for delay. There is an online
support
on unidata's home page and it's more dependable.

I going to address your msg in a general manner until we get to the
same

level. The Grib decoder package was meant to only be a low level API,
the
data resolution that you are looking for would be easier to use a
ToolsUI
program the uses the Grib package. Actually i think you should be able
to
do the refinement w/o doing any coding. The ToolsUI program is located
at:
https://www.unidata.ucar.edu/software/netcdf-java/

It will read in the file and it has options to select areas of data,
the

documentation is a little scarce but intuitive. The is also NCML that
will
let you manipulate the output files.


On the other hand:
The grib package provides a dump, indexer and getData program that can
be
used as templates to get the info you were looking for. There are
javadocs
on all these routines on the decoders page.

There is also a display package IDV that could do the data processing.


Hope this helps.
RObb...











---------- Forwarded message ----------
Date: Tue, 5 Aug 2008 10:59:21 -0400
From: "Bali, Pelin" <address@hidden>
To: address@hidden
Subject: decoders] Grib1 Indexer and GDS info

I am using the Grib1Indexer to read the Grib file. I need to get the
information lat, long, and nxny (145 X 73) as indicated below. I
tried
to use Index.GdsRecord, however I wasn't successful. Although I could
get start points of lat and long, I wasn't able to get neither
resolution nor the end points of lat & long by using gdsRecord. Could
you please show me some that direction on how to get this
information?

Thanks in advance.

------------
rec 98:1554711:date 2008080400 TMP kpds5=11 kpds6=105 kpds7=2
levels=(0,2) grid=16 2 m above gnd 24hr fcst:
  TMP=Temp. [K]
  timerange 0 P1 24 P2 0 TimeU 1  nx 145 ny 73 GDS grid 0 num_in_ave
0
missing 0
  center 1 subcenter 0 process 45 Table 1 scan: WE:SN winds(N/S)
  latlon: lat  -90.000000 to 90.000000 by 2.500000  nxny 10585
          long 0.000000 to 360.000000 by 2.500000, (145 x 73) scan 64
mode 128 bdsgrid 1--


The following is how I use the grib indexer to get the grib data.

RandomAccessFile raf = new RandomAccessFile(filePath, "r");
Grib1Indexer indexer = new Grib1Indexer();
PrintStream ps = new PrintStream(indexPath);
indexer.writeFileIndex(raf, ps, false );


Index index = new Index();
boolean rc = index.open(indexPath);

ArrayList gribIndexRecords = index.getGribRecords();

ArrayList<GdsRecord> gdsRecords = index.getHorizCoordSys();
//
// Build a hashmap for the GDS keys.
//
java.util.HashMap gdsMap = new java.util.HashMap();

for (int a = 0; a < gdsRecords.size(); a++) {
   GdsRecord gdsRecord = gdsRecords.get(a);
   gdsMap.put(gdsRecord.gdsKey, gdsRecord);
}

for (Iterator it = gribIndexRecords.iterator (); it.hasNext (); ) {
   zz++;
GribRecord gribRecord = (GribRecord)it.next();

   if (gribRecord != null) {
       //
       //    Select only temperature.
       //
       if (gribRecord.paramNumber == 11 ) {

           if (gribRecord.levelType1 == 105) {

               if (gribRecord.levelValue1 == 2.0) {

                   if (gribRecord.forecastTime ==
24) {

                       Grib1Data grib1data =
new Grib1Data(raf);

               if (gdsMap.get(gribRecord.gdsKey) !=
null) {
                           GdsRecord gds =
(GdsRecord)gdsMap.get(gribRecord.gdsKey);

System.out.println("Dx: " + gds.dx);

System.out.println("Dy: " + gds.dy);

System.out.println("Grid Shape Code: " + gds.grid_shape_code);

System.out.println("Grid Type: " + gds.grid_type);

System.out.println("La1: " + gds.La1);

System.out.println("LaD: " + gds.LaD);

System.out.println("Latin1: " + gds.latin1);

System.out.println("Latin2: " + gds.latin2);

System.out.println("Lo1: " + gds.Lo1);

System.out.println("LoV: " + gds.LoV);

System.out.println("Major Axis Earth " + gds.major_axis_earth);

System.out.println("Minor Axis Earth " + gds.minor_axis_earth);

System.out.println("Nx: " + gds.nx);

System.out.println("Ny: " + gds.ny);

System.out.println("Radius Spherical Earth: " +
gds.radius_spherical_earth);

System.out.println("Resolution: " + gds.resolution);

System.out.println("Winds: " + gds.winds);
                       }

                       float[] arr =
grib1data.getData(gribRecord.offset1, gribRecord.decimalScale,
gribRecord.bmsExists);

                   } // hour 24
               } // 2m
           } //above ground
       } // end of temperature
   } // grib record is not null
} // end for

Sincerely,

Pelin Bali

WeatherPredict Consulting, Inc.
3200 Atlantic Avenue, Suite 114
Raleigh, NC 27604

Phone (Direct): 919-239-8833
Phone (Main)  : 919-876-3633
Fax           : 919-876-4469

"Imagination will often carry us to worlds that never were.
But without it we go nowhere." - Carl Sagan (Cosmos - 1980)





========================================================================
=======
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric
Research
address@hidden             WWW:
http://www.unidata.ucar.edu/

========================================================================
=======


========================================================================
=======
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric
Research
address@hidden             WWW:
http://www.unidata.ucar.edu/
========================================================================
=======


===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric Research
address@hidden             WWW: http://www.unidata.ucar.edu/
===============================================================================
/*
 * Copyright 1997-2004 Unidata Program Center/University Corporation for
 * Atmospheric Research, P.O. Box 3000, Boulder, CO 80307,
 * address@hidden.
 *
 * This library is free software; you can redistribute it and/or modify it
 * under the terms of the GNU Lesser General Public License as published by
 * the Free Software Foundation; either version 2.1 of the License, or (at
 * your option) any later version.
 *
 * This library is distributed in the hope that it will be useful, but
 * WITHOUT ANY WARRANTY; without even the implied warranty of
 * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU Lesser
 * General Public License for more details.
 *
 * You should have received a copy of the GNU Lesser General Public License
 * along with this library; if not, write to the Free Software Foundation,
 * Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
 */

// $Id: Grib1ExtractRawData.java,v 1.33 2006/04/28 16:19:14 rkambic Exp $


package ucar.grib.grib1;


import ucar.grib.*;
import ucar.unidata.io.RandomAccessFile;
import java.io.IOException;
import java.io.DataOutputStream;

/*
 * Grib1ExtractRawData.java  1.0  09/11/2008
 * @author Robb Kambic
 *
 */

import java.util.*;
import java.util.regex.Matcher;
import java.util.regex.Pattern;


/**
 * A class that scans a GRIB file to extract product information.
 */

public final class Grib1ExtractRawData {

    /*
     * raf for grib file
     */

    /** _more_          */
    private ucar.unidata.io.RandomAccessFile raf = null;

    /*
     * the header of Grib record
     */

    /** _more_          */
    private String header = "GRIB";

    /**
     * Pattern to extract header.
     */
    private static final Pattern productID =
        Pattern.compile("(\\w{6} \\w{4} \\d{6})");

    // *** constructors *******************************************************

    /**
     * Constructs a <tt>Grib1Input</tt> object from a raf.
     *
     * @param raf with GRIB content
     *
     */
    public Grib1ExtractRawData(RandomAccessFile raf) {
        this.raf = raf;
    }

    /**
     * scans a Grib file to gather information that could be used to
     * create an index or dump the metadata contents.
     *
     * @param parmNumber
     * @throws NotSupportedException
     * @throws IOException   if raf does not contain a valid GRIB record
     */
    public final boolean scan( int parmNumber)
            throws NotSupportedException, IOException {
        long start = System.currentTimeMillis();
        // stores the number of times a particular GDS is used
        HashMap                       gdsCounter = new HashMap();
        Grib1ProductDefinitionSection pds        = null;
        Grib1GridDefinitionSection    gds        = null;
        long gdsOffset = 0;
        DataOutputStream dos = new DataOutputStream( System.out );

        //System.out.println("file position =" + raf.getFilePointer()); 
        while (raf.getFilePointer() < raf.length()) {
            if (seekHeader(raf, raf.length())) {
                // Read Section 0 Indicator Section
                Grib1IndicatorSection is = new Grib1IndicatorSection(raf);
                //System.out.println( "Grib record length=" + 
is.getGribLength());
                // EOR (EndOfRecord) calculated so skipping data sections is 
faster
                long EOR = raf.getFilePointer() + is.getGribLength()
                           - is.getLength();
                long SOR = raf.getFilePointer() - is.getLength();

                // skip Grib 2 records in a Grib 1 file
                if( is.getGribEdition() == 2 ) {
                    //System.out.println( "Error Grib 2 record in Grib1 file" ) 
;
                    raf.seek(EOR);
                    continue;
                }

                if( parmNumber == -1 ) { // extract only 1st record
                  raf.seek( SOR );
                  byte[] oneRecord = new byte[ (int)is.getGribLength() ];
                  raf.read( oneRecord );
                  dos.write( oneRecord, 0, oneRecord.length );
                  dos.flush();
                  break;
                }
                long dataOffset = 0;
                try { // catch all exceptions and seek to EOR

                    // Read Section 1 Product Definition Section PDS
                    pds = new Grib1ProductDefinitionSection(raf);
                    if (pds.getLengthErr()) {
                        raf.seek(EOR);
                        continue;
                    }
                    if( parmNumber == pds.getParameterNumber() ) {
                      raf.seek( SOR );
                      byte[] oneRecord = new byte[ (int)is.getGribLength() ];
                      raf.read( oneRecord );
                      dos.write( oneRecord, 0, oneRecord.length );
                      dos.flush();
                    }

                    raf.seek( EOR );
                    continue;

                } catch( Exception e ) {
                    //.println( "Caught Exception scannning record" );
                    e.printStackTrace();
                    raf.seek(EOR);
                    continue;
                }
            }  // end if seekHeader
            //System.out.println( "raf.getFilePointer()=" + 
raf.getFilePointer());
            //System.out.println( "raf.length()=" + raf.length() );
        }  // end while raf.getFilePointer() < raf.length()
        //System.out.println("GribInput: processed in " +
        //   (System.currentTimeMillis()- start) + " milliseconds");
        dos.close();
        return true;
    }  // end scan

    /**
     * Grib edition number 1, 2 or 0 not a Grib file.
     * @throws NotSupportedException
     * @throws IOException
     * @return int 0 not a Grib file, 1 Grib1, 2 Grib2
     */
    public final int getEdition() throws IOException, NotSupportedException {
        int  check  = 0;  // Not a valid Grib file
        long length = (raf.length() < 4000L)
                      ? raf.length()
                      : 4000L;
        if ( !seekHeader(raf, length)) {
            return 0;  // not valid Grib file
        }
        //  Read Section 0 Indicator Section to get Edition number
        Grib1IndicatorSection is = new Grib1IndicatorSection(raf);  // section 0
        return is.getGribEdition();
    }  // end getEdition

    /**
     * _more_
     *
     * @param raf _more_
     * @param stop _more_
     *
     * @return _more_
     *
     * @throws IOException _more_
     */
    private boolean seekHeader(RandomAccessFile raf, long stop)
            throws IOException {
        // seek header
        StringBuffer hdr   = new StringBuffer();
        int          match = 0;

        while (raf.getFilePointer() < stop) {
            // code must be "G" "R" "I" "B"
            char c = (char) raf.read();

            hdr.append((char) c);
            if (c == 'G') {
                match = 1;
            } else if ((c == 'R') && (match == 1)) {
                match = 2;
            } else if ((c == 'I') && (match == 2)) {
                match = 3;
            } else if ((c == 'B') && (match == 3)) {
                match = 4;

                Matcher m = productID.matcher(hdr.toString());
                if (m.find()) {
                    header = m.group(1);
                } else {
                    //header = hdr.toString();
                    header = "GRIB1";
                }
                //System.out.println( "header =" + header.toString() );
                return true;
            } else {
                match = 0;  /* Needed to protect against "GaRaIaB" case. */
            }
        }
        return false;
    }  // end seekHeader



    /**
     * Outputs first record of raw data to STDOUT
     * or parms based on discipline, category, and number
     * @param args filename
     * @throws IOException
     **/
    public static void main(String args[]) 
        throws IOException, NotSupportedException {

      String fileName;
      int number = -1;
      if (args.length == 1) {
        fileName = args[0];
      } else if( args.length == 2) {
        fileName = args[0];
        number = Integer.parseInt( args[1] );

      } else {
        System.out.println( "Not correct number of parms, either 1 or 2" );
        return;
      }  
      RandomAccessFile raf = new RandomAccessFile( fileName, "r" );
      Grib1ExtractRawData erd = new Grib1ExtractRawData( raf );
      erd.scan( number );
  }

}  // end Grib1ExtractRawData