[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [netcdf-java] Help reading grib2 file.



Esteban, the trick with performance is to make your inner loops as efficient as 
possible. So you want to open the GridDataset only once. Im also not seeing 
where you close it, so you have a memory leak there. Also theres no need to put 
final everywhere - that doesnt help anything. Then, get all time points in one 
read, as Robb suggest. So you want something like:

   GridDataset gridDataset = null;
   try {
     gridDataset = GridDataset.open(filename);
     for (Enum parameter : parameters.values()) {

       GridDatatype pwd = gridDataset.findGridDatatype(parameter.name());
       GridCoordSystem pwdGcs = pwd.getCoordinateSystem();

       // Get index value for lat/lon
       int[] idx = pwdGcs.findXYindexFromLatLon(lat, lon, null);

       // Reads values for the given (location,parameter) for all times
       Array data = pwd.readDataSlice(-1, -1, idx[1], idx[0]);
       IndexIterator iter = data.getIndexIterator();
       while (iter.hasNext()) {
         float val = iter.getFloatNext();
       }
     }

   } catch (Exception exc) {
     exc.printStackTrace();

   } finally {
     if (gridDataset != null) {
       try {
         gridDataset.close();
       } catch (IOException ioe) {
         ioe.printStackTrace();
       }
     }
   }


Robb Kambic wrote:
On Sun, 4 Oct 2009, Esteban Wagner wrote:

Robb,

thanks for the code, it worked great!.

I have a question regarding the perfomance of the reading:

With the code you sent me, reading a grib2 file with 11 parameters and
31 times each for only one location takes: 80 seconds.
Is there any way to improve this time?

Esteban,

The best method to improving performance is to limit the number of calls to readDataSlice() even if you have to get more data then you need, you can always reduce it locally. For example, get all the times for a variable by using -1 in the time index position, this should be a huge saving.

Array data = pwd.readDataSlice( -1, -1, idx[1], idx[0]);

As I want to gather values for several locations, say more than 100
locations its a lot of time.(80sec * 100 = 8000 sec = 2.2 hours).

Also, you might want to look at GridCoordSystem methods to see if you can make a bounding box, or some other method to expand your coverage instead of getting just one grid point. Look at makeSubset() method to create a new GridDatatype might be another option.

I would review the javaDocs again, there's probably some other methods that could improve performance.

Robb...



* Perfomance measures were obtained with the following:

* Hardware: Amd Turion 64 X2 TL-58 1.9GHz , 3Gb Ram, Windows Vista 32,
5400Rpm HD.
* Grib2 file: ftp://polar.ncep.noaa.gov/pub/waves/develop/multi_1.latest_run/multi_1.glo_30m.all.grb2
* Library versions:(NetCDF:Version 4.0.16; Grib:Version 6.0.16)
* Code :
---------------
import java.io.File;
import java.util.Date;

import ucar.ma2.Array;
import ucar.ma2.IndexIterator;
import ucar.nc2.dt.GridCoordSystem;
import ucar.nc2.dt.GridDatatype;
import ucar.nc2.dt.grid.GridDataset;

public class ReadLatLonFromGrib {
    public static void main(final String[] args) {
        final File file = new File(
                "src/test/resources/multi_1.glo_30m.all.grb2");
        final float lat = 39;
        final float lon = 179;
        final long startTime = new Date().getTime();
        // Read the values for the given location
        readValuesForLocation(file, lat, lon);

        final long endTime = new Date().getTime();
        // Calculate reading elapsed time(seconds).
        final long elapsedTime = (endTime - startTime) / 1000;
        System.out.println("Time elapsed for reading "
                + parameters.values().length
+ " parameters, for 31 times, in one location: " + elapsedTime
                + " secs.");
    }

    private static enum parameters {
        Direction_of_swell_waves, Direction_of_wind_waves,
Mean_period_of_swell_waves, Mean_period_of_wind_waves,
Primary_wave_direction, Primary_wave_mean_period,
Significant_height_of_combined_wind_waves_and_swell,
Significant_height_of_swell_waves, Significant_height_of_wind_waves,
Wind_direction_from_which_blowing, Wind_speed
    };

private static void readValuesForLocation(final File file, final float lat,
            final float lon) {
System.out.println("Reading grib values for lat: " + lat + " and lon: "
                + lon);
        try {
            // For each parameter
            for (final Enum parameter : parameters.values()) {
                final GridDataset gridDataSet = GridDataset.open(file
                        .getAbsolutePath());
                // For each time
                for (int i = 0; i < 31; i++) {
                    final GridDatatype pwd = gridDataSet
                            .findGridDatatype(parameter.name());
final GridCoordSystem pwdGcs = pwd.getCoordinateSystem();
                    final int[] result = null;
                    // Get index value for lat/lon
final int[] idx = pwdGcs.findXYindexFromLatLon(lat, lon,
                            result);
// Reads values for the given (location,parameter,time) final Array data = pwd.readDataSlice(i, -1, idx[1], idx[0]);
                    final IndexIterator iter = data.getIndexIterator();
                    while (iter.hasNext()) {
                        // Obtain the value
                        final float val = iter.getFloatNext();
                    }
                }
            }
            System.out.println("Finish reading values");
        } catch (final Exception exc) {
            exc.printStackTrace();
        }
    }
}

Thanks.
Regards.
Esteban.
-------------------------------------------------------------------------------------------------


On Fri, Oct 2, 2009 at 4:14 PM, Robb Kambic <address@hidden> wrote:
On Tue, 29 Sep 2009, Esteban Wagner wrote:

Hi,
I have a grb2 file containing global Primary_wave_direction forecasts. I
would like to read the
values it contains.

By the moment I was able to read the file and obtain a GridDataset using
the following code:
------------
(NetCDF:Version 4.0.16; Grib:Version 6.0.16)

final File file = new
File("src/test/resources/multi_1.glo_30m.DIRPW.grb2");
final GridDataset gridDataSet = GridDataset.open(file.getAbsolutePath());
----------
the file can be downloaded from:

ftp://polar.ncep.noaa.gov/pub/waves/develop/multi_1.latest_run/multi_1.glo_30m.DIRPW.grb2
----------
Now I would like if someone can tell me which are the methods from
gridDataSet I should use in order
to obtain the different forecast values of the Primary_wave_direction for
a given grid point.

Thanks,

Regards.

Esteban.


Esteban,

I'll reference the JavaDocs first:

http://www.unidata.ucar.edu/software/netcdf-java/v4.1/javadoc/ucar/nc2/dt/GridDataset.html http://www.unidata.ucar.edu/software/netcdf-java/v4.1/javadoc/ucar/nc2/dt/GridDatatype.html http://www.unidata.ucar.edu/software/netcdf-java/v4.1/javadoc/ucar/nc2/dt/GridCoordSystem.html

I'm assuming by grid point you mean Lat/Lon
One needs to be at the GridCoordSystem level to be able to convert the
Lat/Lon point into indices. Then  use the indices information in the
GridDatatype readDatSlice routine to get the value. Here's a short sample
program.

public class GetDataFromLatLon {
 public static void main(String[] args) {

      try {

       final File file = new
File("C:/data/grib/multi_1.glo_30m.DIRPW.grb2");
       final GridDataset gridDataSet =
GridDataset.open(file.getAbsolutePath());
       GridDatatype pwd = gridDataSet.findGridDatatype(
"Primary_wave_direction");
       System.out.println( pwd.getInfo());

       GridCoordSystem pwdGcs = pwd.getCoordinateSystem();
       int[] result = null;
       // Get index value for Lat 30.0 and Lon 179
       int[] idx = pwdGcs.findXYindexFromLatLon( 30.0, 179.0, result );
       //int[] idx = pwdGcs.findXYindexFromLatLon( 90.0, 0.0, result );
// Extract data value for time 0, no Vert index, Lat index, Lon index
       Array data = pwd.readDataSlice( 0, -1, idx[1], idx[0]);
       // Another option: read all data values for timeIndex = 0
       //Array data = pwd.readVolumeData( 0 );
       IndexIterator iter = data.getIndexIterator();
       while(iter.hasNext()) {
               float val = iter.getFloatNext();
               System.out.println( "Primary_wave_direction = "+ val);
       }

       System.out.println( "Success");
      } catch (Exception exc) {
             exc.printStackTrace();
      }
   }
}
===============================================================================
Robb Kambic                                Unidata Program Center
Software Engineer III                      Univ. Corp for Atmospheric
Research
address@hidden WWW: http://www.unidata.ucar.edu/ ===============================================================================


===============================================================================
Robb Kambic                       Unidata Program Center
Software Engineer III               Univ. Corp for Atmospheric Research
address@hidden           WWW: http://www.unidata.ucar.edu/
===============================================================================


------------------------------------------------------------------------

_______________________________________________
netcdf-java mailing list
address@hidden
For list information or to unsubscribe, visit: http://www.unidata.ucar.edu/mailing_lists/