Re: [netcdfgroup] (no subject)

Subject: Re: [netcdfgroup] (no subject)
From: Dennis Heimbigner <dmh@xxxxxxxxxxxxxxxx>
Date: Tue, 19 Feb 2013 20:31:32 -0700

You should upgrade to 4.2.1.1

Taylor Binnington wrote:

Thank you both.

First, in reply to Christopher's suggestion:

I've explored the documentation
here<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/index.html>
as
well as the 
test_var.cpp<http://www.unidata.ucar.edu/software/netcdf/docs/cxx4/test_var_8cpp-example.html#_a10>
example
program, in addition to a few other sources. I'm finding the documentation
quite opaque, regarding learning to extract only certain indices or an
array via the NetCDF API. However, one thing I've gleaned is that
subsetting parameters are specified as a NetCDF Group; however, this data
structure is not found in MERRA files, since Groups are only supported by
NetCDF-4 files.

Second, in reply to Dennis' request. The output from

ncdump  '[log][show=fetch]
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf?U50M_EOSGRID_Data_Fields[0:23][282:282][441:441]
'

is the following:

Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds

Log:fetch complete: 2.906 secs
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.das

Log:fetch complete: 0.443 secs
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dds?U50M_EOSGRID_Data_Fields[0:23][282][441]

Log:fetch complete: 0.228 secs
Log:prefetch: U50M_EOSGRID_Data_Fields=24
Log:prefetch: U50M_EOSGRID_Data_Fields
Log:prefetch.final: U50M_EOSGRID_Data_Fields*[0:23/24][0:360/361][0:539/540]
Log:fetch:
http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf.dods?U50M_EOSGRID_Data_Fields

Log:fetch complete: 58.994 secs
Log:prefetch.complete
Log:prefetch.vars:  U50M_EOSGRID_Data_Fields

netcdf MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101 {
dimensions:
        TIME_EOSGRID = 24 ;
        XDim_EOSGRID = 1 ;
        YDim_EOSGRID = 1 ;
variables:
        float U50M_EOSGRID_Data_Fields(TIME_EOSGRID, YDim_EOSGRID,
XDim_EOSGRID) ;
                U50M_EOSGRID_Data_Fields:_FillValue = 1.e+15f ;
                U50M_EOSGRID_Data_Fields:long_name = "Eastward wind at 50 m
above surface" ;
                U50M_EOSGRID_Data_Fields:standard_name =
"eastward_wind_at_50_meters" ;
                U50M_EOSGRID_Data_Fields:units = "m/s" ;
                U50M_EOSGRID_Data_Fields:scale_factor = 1.f ;
                U50M_EOSGRID_Data_Fields:add_offset = 0.f ;
                U50M_EOSGRID_Data_Fields:missing_value = 1.e+15f ;
                U50M_EOSGRID_Data_Fields:fmissing_value = 1.e+15f ;
                U50M_EOSGRID_Data_Fields:vmin = -1.e+30f ;
                U50M_EOSGRID_Data_Fields:vmax = 1.e+30f ;
                U50M_EOSGRID_Data_Fields:valid_range = -1.e+30f, 1.e+30f ;
                U50M_EOSGRID_Data_Fields:coordinates = "TIME_EOSGRID
YDim_EOSGRID XDim_EOSGRID" ;

// global attributes:
                :HDFEOSVersion = "HDFEOS_V2.14" ;
                :missing_value = 1.e+15f ;
                :Conventions = "CF-1.0" ;
                :title = "MERRA reanalysis. GEOS-5.2.0" ;
                :history = "File written by CFIO" ;
                :institution = "Global Modeling and Assimilation Office,
NASA Goddard Space Flight Center, Greenbelt, MD 20771" ;
                :source = "Global Modeling and Assimilation Office.
GEOSops_5_2_0" ;
                :references = "http://gmao.gsfc.nasa.gov/research/merra/"; ;
                :comment = "GEOS-5.2.0" ;
                :contact = "http://gmao.gsfc.nasa.gov/"; ;
data:

 U50M_EOSGRID_Data_Fields =
  5.483779,
  4.986389,
  4.076741,
  3.479807,
  3.125561,
  2.818876,
  2.536896,
  2.388737,
  2.501732,
  2.531001,
  2.440533,
  2.245501,
  1.99101,
  1.798543,
  1.793866,
  1.694912,
  1.561274,
  1.471124,
  1.408249,
  1.449105,
  1.588835,
  1.835351,
  2.13877,
  2.48236 ;
}



On Tue, Feb 19, 2013 at 11:00 AM, Lynnes, Christopher S. (GSFC-6102) <
christopher.s.lynnes@xxxxxxxx> wrote:

On Feb 19, 2013, at 10:48 AM, Taylor Binnington <tbinnington@xxxxxxxxx>
wrote:

Thank you Dennis.
I'm using version 4.2.1, netcdf-cxx4 by Lynton Appel, on openSUSE 12.1,

so I think it's fairly up-to-date. The problem is not that I am unable to
open the file (I, too, can print out the 24 values), but that doing so
apparently uses ~19 mb of bandwidth, according to the iftop monitoring
utility. This seems quite excessive.

Please let me know if there is specific information that I should

provide here. Much appreciated,

Taylor

Taylor,
  I think Benno might have put his finger on the issue over on the
opendap-tech mailing list.  That is, instead of passing the fully
constrained URL to a read function, open the URL without the constraints
and then use one of the methods that allows you to specify the start and
length (or corner points) of the segment you want to retrieve.  I know they
must be there somewhere, because the C API certainly has it.


On Mon, Feb 18, 2013 at 5:57 PM, Dennis Heimbigner <dmh@xxxxxxxxxxxxxxxx>

wrote:

The opendap code in the netcdf-C libraries
(used by the C++ library) will in fact
properly handle the constraint you specified.
I just ran it and as expected, it is only pulling
in the 24 values you requested (took some 5 seconds to
obtain and print out the data).

One possibility is that you are using a rather old version
of the netcdf-C library.

=Dennis Heimbigner
 Unidata

Taylor Binnington wrote:
Hello,

I have an 'understanding' type of question. I have omitted details of my
OS, NetCDF installation, and a minimal example since I do not think they
are important for my question, but I can provide these details if they

are

important.

When I attempt to open a MERRA file using the NetCDF C++ libraries, with

line like the following, in my code:

NcFile datafile("

http://goldsmr2.sci.gsfc.nasa.gov/opendap/hyrax/MERRA/MAT1NXSLV.5.2.0/1990/01/MERRA100.prod.assim.tavg1_2d_slv_Nx.19900101.hdf?U50M_EOSGRID_Data_Fields[0:23][282:282][441:441]

",NcFile::read);

...I notice that quite a lot (nearly 20 MB) is being transferred from

Hyrax

to my PC. However, all that I am interested in are the actual array

values

(24 separate 32-bit real numbers) of the U50M_EOSGRID_Data_Fields. It has
been suggested that the entire array (of size 24*361*540 32-bit reals) is
being transferred by NetCDF before any subsetting takes place, is this
true? If so, is it then not possible to use OPeNDAP URLs to subset
on-the-fly through the NetCDF libraries?

Thank you for your help!
Taylor



------------------------------------------------------------------------

_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:

http://www.unidata.ucar.edu/mailing_lists/



--
Taylor Binnington
e. tbinnington@xxxxxxxxx
c. 647 926 4144
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe,  visit:

http://www.unidata.ucar.edu/mailing_lists/

--
Dr. Christopher Lynnes, NASA/GSFC, ph: 301-614-5185

References:
- [netcdfgroup] (no subject)
  - From: Taylor Binnington
- Re: [netcdfgroup] (no subject)
  - From: Dennis Heimbigner
- Re: [netcdfgroup] (no subject)
  - From: Taylor Binnington
- Re: [netcdfgroup] (no subject)
  - From: Lynnes, Christopher S. (GSFC-6102)
- Re: [netcdfgroup] (no subject)
  - From: Taylor Binnington

2013 messages navigation, sorted by:
1. Thread
2. Subject
3. Author
4. Date
5. ↑ Table Of Contents
Search the netcdfgroup archives: