[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #WVT-997700]: dap access to huge files fails

> So am I right, the thredds delivers classic format, even when the
> aggregated files are netcdf-4? 

Correct. The reason is that when you use dodsC, you are using
what is called the DAP2 protocol. The thredds server converts
the input file(s) -- the aggregated netcdf4 files in this case
-- to the DAP2 format and sends the converted file to the client
program. On the client, the netcdf-c library converts DAP2 to
something it can use: netcdf classic in this case.  So no matter
what the original file was on the thredds server, it will be
converted to a netcdf classic file on the client.

The thredds server can be made to use a different protocol
called DAP4 that the client will convert from DAP4 to netcdf-4
(enhanced). However, looking at the catalog entry, I see that
DAP4 was not enabled on the thredds server.

As for netcdf-4.4.1 vs the latest version. I am curious.
Did you successfully download the data (say variable lap_fric_u)
as well as print the metadata?


> Dennis,
> many thanks for taking the time. I confess, the data set is really huge.
> It has a lot a variables, not only grid points.
> 
> First, please use
> 
> http://thredds-iow.io-warnemuende.de/thredds/dodsC/genus/IOW-THREDDS-genus_exp19_run3_5day_ocean_2017-10-20-16.nc
> 
> for testing (note run3 instead of run2). In the run2-dataset one
> aggregated file seems to be in error. Since all ferret plots look fine
> anyway, difficult to identify. run3 is fully correct, but the error
> persists. Sorry for the confusion.
> 
> Indeed,
> ncdump -k
> http://thredds-iow.io-warnemuende.de/thredds/dodsC/genus/IOW-THREDDS-genus_exp19_run3_5day_ocean_2017-10-20-16.nc
> results in "classic"
> 
> Direct ncdump on one file aggregated into the data set gives
> 
> ncdump -k ocean_day.nc4
> netCDF-4
> 
> So am I right, the thredds delivers classic format, even when the
> aggregated files are netcdf-4? But see below - why does the netcdf
> 4.4.1.1-library work and the 4.6.1 does not?
> 
> The html catalog entry is
> https://thredds-iow.io-warnemuende.de/thredds/projects/genus/catalog_exp19_run3.html
> Please use the second data set
> https://thredds-iow.io-warnemuende.de/thredds/projects/genus/catalog_exp19_run3.html?dataset=IOW-THREDDS-genus_exp19_run3_5day_ocean_2017-10-20-16
> 
> The client machine, where the netcdf-library was build on and where
> ncdump fails on is a 64-bit machine:
> 
> <PHY-50:mschmidt>uname -a
> Linux PHY-50 4.12.14-lp150.12.4-default #1 SMP Tue May 22 05:17:22 UTC
> 2018 (66b2eda) x86_64 x86_64 x86_64 GNU/Linux
> 
> The machine where the thredds server lives on is:
> <phy-8:mschmidt>uname -a
> Linux phy-8 4.4.138-59-default #1 SMP Mon Jun 18 13:48:42 UTC 2018
> (f0b8f6b) x86_64 x86_64 x86_64 GNU/Linux
> 
> The tredds-catalog in question is attached.  It is the second entry:
> ocean data, 5-day time averages
> 
> Symptoms that may be of some help:
> 
> - confusing for me is that the access works well with a netcdf
> 4.4.1.1-library build on the same machine a few hours earlier using the
> same build-script, only filenames are changed. Attached. Also config.log
> from building the 4.4.1.1 lib and from building the latest lib are
> attached. The resulting "Defines" differ a lot, but I do not understand
> details.
> #define ENABLE_DAP4 1 in the new version. The result of ncdump ist
> classic for the 4.4.1.1 library. Ferret build with this library works well.
> 
> - ncdump build with the latest netcdf-library works well on a smaller
> data set. It has the same structure, but includes less time steps.
> ncdump
> http://thredds-iow.io-warnemuende.de/thredds/dodsC/genus/IOW-THREDDS-genus_exp19_run1_5day_ocean_2017-10-20-16.nc
> (Note run1 instead of run3. The catalog entry is almost the same except
> filenames.)
> 
> Best,
> Martin Schmidt
> 
> On 07/02/2018 06:14 PM, Unidata netCDF Support wrote:
> > Also, have you tried doing this on a 64-bit machine?
> >
> >> The problem appears to be that at least the variable
> >> Float32 lap_fric_u[time = 1776][st_ocean = 89][yu_ocean = 382][xu_ocean = 
> >> 273];
> >> is too large to fit into a netcdf-3 variable.
> >> Remember that the netcdf-c library attempts to translate
> >> an opendap (i.e. DAP2) request into an equivalent netcdf-3
> >> representation so that it can be accessed thru the netcdf API.
> >> The lap_fric_u variable size is 1776*89*382*273*4bytes = 65935449216bytes
> >> which is too big for netcdf-3.
> >> Can you send me the a url to the catalog entry on the thredds server
> >> for this dataset? If by some chance they have DAP4 enabled, then
> >> you should be able to at least get ncdump -h to work.
> >>
> >>> as usual letters come if something does not work. Sorry. I had reported
> >>> this error before, the ticket was closed as fixed. But the error still
> >>> persists, also in the very latest code downloaded from the git. You may
> >>> want to know this.
> >>>
> >>> The version is netcdf library version 4.6.2-development of Jul  1 2018
> >>> 21:41:57  After building with autotools and configure all tests pass.
> >>>
> >>> The error symptom is:
> >>>
> >>> ./ncdump
> >>> http://thredds-iow.io-warnemuende.de/thredds/dodsC/genus/IOW-THREDDS-genus_exp19_run2_5day_ocean_2017-10-20-16.nc
> >>>
> >>>
> >>> fails, when the header section is read.
> >>>
> >>> .........
> >>>
> >>> data:
> >>>
> >>> NetCDF: Index exceeds dimension bound
> >>> Location: file vardata.c; line 473
> >>>
> >>> Linking the library to ferret, coordinates are reported as out of order.
> >>> Trying to print them, kills ferret. With smaller data sets also the
> >>> latest netcdf works well.
> >>>
> >>> Linking this ferret version with netcdf-4.4.1.1 instead (same hdf5 and
> >>> szip) everything is fine. Also ncdump works well with this library.
> >>>
> >>> The tredds is public, so you may reproduce the error by yourself. Hence,
> >>> I do not add a complete error report yet. If you do not see the error,
> >>> please let me know, I can deliver all details on the netcdf-build and
> >>> the thredds as well.
> >>>
> >>> Greetings and sorry for coming with the same issue again,
> >>>
> >>> Martin Schmidt
> >>>
> >>>
> >>>
> >>>
> >> =Dennis Heimbigner
> >> Unidata
> >>
> > =Dennis Heimbigner
> >    Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: WVT-997700
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> 
> --
> Dr. Martin Schmidt
> Leibniz-Institute for Baltic Sea Research
> Seestrasse 15
> D18119 Rostock
> Germany
> 
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: WVT-997700
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.