[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #IXE-857786]: C++ NetCDF Library Troubles

> http://tds.hycom.org/thredds/GLBv0.08/expt_93.0.html?dataset=GLBv0.08-expt_93.0
This is the one I wanted to see. It allowed me to see if
there was a reasonable way to download the data using some other
protocol than DAP2 (.../dodsC/...), specifically DAP4.
Unfortunately, there is no good alternative.

So, I think your only solution is to download the variables in pieces.

> Thank you for your response.  I’m afraid I’m not sure to which URL you are 
> referring?
> 
> This is the THREDDS URL I am using in my code:
> http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0.html 
> <http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0.html>
> 
> And this URL leads to a more general page with different data access options:
> http://tds.hycom.org/thredds/GLBv0.08/expt_93.0.html?dataset=GLBv0.08-expt_93.0
>  
> <http://tds.hycom.org/thredds/GLBv0.08/expt_93.0.html?dataset=GLBv0.08-expt_93.0>
> 
> Hope this helps,
> 
> Cheers,
> Blake
> 
> > On Jun 4, 2018, at 7:39 PM, Unidata netCDF Support <address@hidden> wrote:
> >
> > The problem is the size of some of the variables is very large: especially
> > salinity.
> > Since you are  using the DAP2 protocol (your url is .../dodsC/...)
> > which maps to netcdf-3. Those variables are too large for netcdf-3.
> >
> > Can you tell me the url for the catalog entry for this dataset?
> >
> >
> >>
> >> Using the utilities provided with the core C library (ncdump, nccopy), it 
> >> appears that there is a problem with the remote netCDF file.  
> >> Specifically, it appears that the data contained in some of the variables 
> >> violates size constraints for the variables.  You can see this data is 
> >> using the `classic` netCDF3 file format via:
> >>
> >>     $ ./ncdump -k http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0
> >>
> >> We can view the metadata for the file using:
> >>
> >>     $ ./ncdump -h http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0
> >>
> >> Normally, we can use `ncdump -v [variable name]` to look at individual 
> >> variables.  Unfortunately, we get index bounds errors when we attempt to 
> >> do this here.  There is a workaround, however.  I was able to use `nccopy 
> >> -7 http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0 tmp.nc` to copy 
> >> the data into a `netCDF-4 classic model` file.  This operation took a 
> >> little time and the resultant file is 33GB in size.  However, I am able to 
> >> query the file using ncdump and get the variable names, as one would 
> >> expect.
> >>
> >> Once I was able to get to this step, I was able to investigate the 
> >> variables `salinity` and `water_temp`.  It looks like the majority of the 
> >> data stored in `salinity` are 0, and all of the values store for 
> >> `water_temp` are 0, suggesting no observations were recorded.  While not 
> >> exhaustive, I checked some of the data through the TDS web interface (see 
> >> http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0.ascii?water_temp[0:2:0][0:1:0][0:1:0][0:1:0]
> >>  as an example).  There, we can see that the water_temp is `-30000`, which 
> >> corresponds to the default fill value for this data, suggesting no 
> >> observation was recorded, and that the issue is not in the netCDF library. 
> >> Why this is the case, I cannot say.  But it does show that there is 
> >> nothing wrong with the program you have written.
> >>
> >> I hope this helps; I'm sorry I couldn't provide an immediate solution, but 
> >> it should at least assure you that the issue isn't in the code you have 
> >> written.
> >>
> >> - Ward
> >> On May 31, 2018, 8:42 PM -0600, Blake Ian Barry Cole <address@hidden>, 
> >> wrote:
> >>> New Ticket: C++ NetCDF Library Troubles
> >>>
> >>> Good afternoon,
> >>>
> >>> I feel I am very close to implementing a functional c++ program to 
> >>> extract a chunk of NetCDF data from the HYCOM/GOFS 3.1 Hindcast data.
> >>> [http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0.html 
> >>> <http://tds.hycom.org/thredds/dodsC/GLBv0.08/expt_93.0.html>]
> >>>
> >>> However, when I attempt to pass an array (of type float, or double, say) 
> >>> by reference to getVar(), the temperature and salinity values are not 
> >>> being written to the array.
> >>>
> >>> I believe I have sized/pre-allocated the array correctly, and am using 
> >>> the startp and countp vectors correctly; I do not know what is going 
> >>> wrong. This is a typical output of my program:
> >>> -----------------------
> >>> NETCDF FILE INFO:
> >>> 14 variables
> >>> 10 attributes
> >>> 4 dimensions
> >>> 0 groups
> >>> 0 types
> >>>
> >>> -----------------------
> >>> VARIABLES:
> >>> depth
> >>> lat
> >>> lon
> >>> salinity
> >>> salinity_bottom
> >>> surf_el
> >>> tau
> >>> time
> >>> water_temp
> >>> water_temp_bottom
> >>> water_u
> >>> water_u_bottom
> >>> water_v
> >>> water_v_bottom
> >>>
> >>> -----------------------
> >>> DIMENSIONS:
> >>> depth
> >>> lat
> >>> lon
> >>> time
> >>>
> >>> -----------------------
> >>> DIMENSION SIZES:
> >>> depth = [0:5000] [n=40]
> >>> lat = [-80:90] [n=3251]
> >>> lon = [0:359.92] [n=4500]
> >>> time = [157812:161409] [n=1168]
> >>>
> >>> -----------------------
> >>> SPATIAL RANGE:
> >>> Requested:
> >>> LAT = [0:2]
> >>> LON = [154:155]
> >>> Actual:
> >>> LAT[1500:1525] = [0:2]
> >>> LON[1925:1938] = [154:155.04]
> >>>
> >>> startp[0] = 1167
> >>> countp[0] = 1
> >>> startp[1] = 0
> >>> countp[1] = 40
> >>> startp[2] = 1500
> >>> countp[2] = 26
> >>> startp[3] = 1925
> >>> countp[3] = 14
> >>>
> >>> --------------------------------
> >>> ARRAYS CREATED:
> >>> SALT[40][26][14]
> >>> TEMP[40][26][14]
> >>>
> >>> -----------------------
> >>> SALT[31][3][6] = 161409 psu
> >>> SALT[33][0][1] = 1.76622e+264 psu
> >>> SALT[33][9][0] = 1.76622e+264 psu
> >>> SALT[33][13][12] = 1.76622e+264 psu
> >>> SALT[33][17][6] = 1.76622e+264 psu
> >>> SALT[33][20][8] = 9.75388e+199 psu
> >>> SALT[33][20][9] = 7.62831e+228 psu
> >>> SALT[33][20][10] = 7.96474e+175 psu
> >>> SALT[33][20][12] = 5.13122e+252 psu
> >>> SALT[33][21][10] = 1.76622e+264 psu
> >>> SALT[34][6][2] = 1.76622e+264 psu
> >>> SALT[34][6][10] = 1.76622e+264 psu
> >>> SALT[34][8][12] = 359.92 psu
> >>> SALT[37][2][8] = 1.76622e+264 psu
> >>> SALT[37][11][8] = 1.76622e+264 psu
> >>> SALT[37][12][2] = 1.76622e+264 psu
> >>> SALT[39][3][11] = 155.04 psu
> >>> SALT[39][4][12] = 155.04 psu
> >>> SALT[39][5][13] = 155.04 psu
> >>> SALT[39][7][0] = 155.04 psu
> >>> SALT[39][16][2] = 1.76622e+264 psu
> >>> SALT[39][23][8] = 1.76622e+264 psu
> >>> SALT[39][24][8] = 1.76622e+264 psu
> >>> *** SUCCESS! ***
> >>>
> >>>
> >>> where the first index refers to depth, the second to latitude, and the 
> >>> third to longitude.
> >>>
> >>> Are there any obvious “gotchas” that new NetCDF users typically get wrong?
> >>> I will attach my current code for reference. Thank you in advance for any 
> >>> help you are able to give!
> >>>
> >>> Cheers,
> >>>
> >>> Blake Cole
> >>> Phd Student
> >>> Mechanical Engineering Dep’t, MIT
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Ticket Details
> >>> ===================
> >>> Ticket ID: IXE-857786
> >>> Department: Support netCDF
> >>> Priority: Normal
> >>> Status: Open
> >>> Link: 
> >>> https://andy.unidata.ucar.edu/esupport/staff/index.php?_m=tickets&_a=viewticket&ticketid=29417===================
> >>> NOTE: All email exchanges with Unidata User Support are recorded in the 
> >>> Unidata inquiry tracking system and then made publicly available through 
> >>> the web. If you do not want to have your interactions made available in 
> >>> this way, you must let us know in each email you send to us.
> >>
> >>
> >
> > =Dennis Heimbigner
> >  Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: IXE-857786
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: IXE-857786
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.