[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #JZZ-916731]: problems in building netcdf 4.6.0

There is a problem. I do not believe we can accept patches
because of potential copyright issues. Either you need to explicitly
renounce all copyright on the patches or (better) create a pull request
on github containing the proposed changes.

Is there any relatively simple characterization of the differences
between vis.studio and mingw with respect to, say, headers?
There is, as a recall one place where we do make the distinction
because legal files paths for mingw can differ from those accepted
by vis.studio. I think it is, as a rule, best to include/exclude
a header by detecting if it is present using the configure.ac macros
and having a flag in config.h. However, I admit to not always following
this rule.

Unfortunately, I am not in a position to search the internals of MatLab
to see how it uses the DOD {} construct. So, I guess this is is going
to be a case-by-case set of fixes over time. In the mean-time,
you can force a specific strlen by suffixing your url
with '#stringlength=nnn' or (for a specific variable)
'#stringlength_var=nnn' where 'var' is the fully qualified
name of the variable



> Thanks for your reply. With respect to the patches: I do not want to hurry 
> you up, just wanted to inform you about what I have been doing, hoping that 
> it will also be useful for other developments. I am now making an inventory 
> of all places in the software where the switch _MSC_VER is used, checking 
> whether the switch is used for a compiler-dependent or rather operating 
> system dependent choice. In the latter case, it would be better to replace it 
> by _WIN32, which will also cover Microsoft C compiler but is generalizable to 
> MINGW MSYS2. If I find the need for any other patches, I will let you know.
> 
> 
> 
> With respect to the unreadable file: I am not a specialist at all on DODS, 
> and I cannot help you with a clear description of the protocol. In fact, it 
> is my impression that the use of the data attributes (in the DAS output) to 
> store string length is not entirely orthodox with respect to the DODS 
> protocol. It says (on opendap.org) that the DAS file is used to transmit data 
> attributes that can be anything but size and structure of the variables, 
> which should go in the DDS. However, the DDS apparently has no place holder 
> for string length of individual variables, so it looks as if libraries (such 
> as the Matlab library that was used to produce this particular problem file) 
> abuse DAS to store information on string length of individual variables that 
> they cannot get rid of in DDS.
> 
> My advice would be to have a look at how exactly Matlab does this, as Matlab 
> is probably the dominant library used to produce this type of files. Or to 
> use Matlab to produce a variety of openDAP files, and then check whether they 
> are readable. In the end, apart from the standards, it may be best to take a 
> pragmatic view. If this is how people do it in actual files stored on openDAP 
> servers, it is better to make sure one can read it...
> 
> 
> 
> Best wishes,
> 
> Peter
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> Peter Herman
> 
> T
> 
> +31(0)88335 7862
> 
> E
> 
> address@hidden
> 
> 
> 
> 
> [Logo]<http://www.deltares.com/>
> 
> www.deltares.com<http://www.deltares.com/>
> 
> Postbus 177
> 2600 MH Delft
> 
> [Deltares Twitter]
> <http://www.twitter.com/deltares>
> 
> [Deltares LinkedIn]<http://www.linkedin.com/company/217430>
> 
> [Deltares Facebook]<https://www.facebook.com/pages/Deltares/154189334634001>
> 
> 
> 
> [Think before printing]Please consider the environment before printing this 
> email
> 
> 
> -----Original Message-----
> From: Unidata netCDF Support [mailto:address@hidden]
> Sent: maandag 12 maart 2018 22:14
> To: Peter Herman
> Cc: address@hidden; Peter Herman
> Subject: [netCDF #JZZ-916731]: problems in building netcdf 4.6.0
> 
> Thank you for the patches. It may take us a while to work thru them.
> You say:
> > Although these dimensions are given in the dds, the netcdf opendap
> > software assumes default string length of 64 and neglects the given
> > string lengths.
> 
> The problem is that I could never find a defined standard about things like 
> DODS attribute groups and where they can occur.
> The netcdf-c code will translate and use DODS { Int32 strlen 211; } if it is 
> at the top level of the .das file but (currently) not anywhere else.
> When the code was written (a decade ago?) and if my memory is correct, the 
> only examples I could find had only a global DODS attribute group.
> 
> If you can point me to such a standard describing DODS (and other groups such 
> as NC_GLOBAL) I would be happy to implement it.
> In the meantime, your usage seems reasonable, so I will add a github issue to 
> implement the DODS {} attribute group on individual variables, but I cannot 
> promise when I will get to it.
> 
> >
> > In previous emails I explained that I am trying to build netcdf with 
> > opendap capabilities in MINGW/MSYS2, because I want to use the library in a 
> > package of the data analysis software R. I had several problems that I 
> > solved sequentially in the following way:
> >
> >
> >
> > 1.     Some adaptations to CMake were needed - patch 1, copied from the 
> > MSYS2 system, but addition of an extra measure (make mkstemp unavailable, 
> > because it generated problems in compiling the R package)
> >
> > 2.     Typing error correction in dutil.c
> >
> > 3.     Export strlcat to the netcdf.dll, so that it is available for 
> > subroutines that normally cannot use it because in the standard set-up it 
> > is not included into the dll
> >
> > 4.     Change ddispatch.c such that it uses the Windows TEMP environmental 
> > variable to make a temp file
> >
> >
> >
> > I explained these different patches in my previous e-mails. However, I 
> > still had problems reading certain opendap files. The reason for this was 
> > that these files used NaN as fill_value, and the netcdf routine dapcvt 
> > could not handle this exception. In order to detect this, I had to be able 
> > to switch on the log/debug mode of opendap, which I did in patch no. 5. I 
> > subsequently patched the routine dapcvt.c so that it no longer chokes on 
> > NaN. However, I feel my solution is not very elegant and I am not sure that 
> > it will always be consistent. I include the patches also in this file.
> >
> >
> >
> > 5.     (a non-patch) I use this to switch debug logging on or off
> >
> > 6.     A patch of dapcvt.c so that it converts strings to numbers correctly 
> > when using the Windows run-time libraries (which sscanf does by default), 
> > and so that it is resistant to NaN entries. The patch works for NaN in 
> > double fields, but is not tested on other cases.
> >
> >
> >
> > Question: I remarked that elsewhere in the netcdf source files there are 
> > routines that properly handle NaN, Inf etc., as well as convert strings to 
> > certain types (e.g. long integers). I wondered whether these routines could 
> > be used here too? I have the impression that this could be more resilient 
> > than the solution presented by the present patch, and may not require much 
> > additional programming. I would be interested in your response to this 
> > suggestion.
> >
> >
> >
> > As for now, I am able to read all openDAP files that were on my list,
> > with one exception. The file
> > http://opendap.deltares.nl/thredds/dodsC/opendap/rijkswaterstaat/water
> > base/catalog.nc does not read in correctly. However, this is not
> > related to my build, as it also occurs when I use the ncdump build
> > under Linux (Ubuntu). Maybe this file is badly constructed, or there
> > is a bug in the opendap software that makes it impossible to read.
> > Note that the netcdf file that was used by the openDAP server is
> > readable, however. The problem is in the translation of the dimensions
> > of the character arrays. Although these dimensions are given in the
> > dds, the netcdf opendap software assumes default string length of 64
> > and neglects the given string lengths. I do not know why this is. I
> > include the files ncdumpubuntu.txt for the output of using ncdump with
> > the openDAP file on Ubuntu, ncdumpWindows.txt for the same operation
> > with my build of ncdump on Windows, and ncdumpcatalognc.txt for the ou
> tput of using ncdump (on Windows) directly on the netcdf file, so not through 
> openDAP. This contains the 'true' structure of the file.
> >
> >
> >
> > I am looking forward to your reaction
> >
> >
> >
> >
> >
> > With kind regards,
> >
> > Peter
> >
> >
> >
> >
> > Peter Herman
> >
> > T
> >
> > +31(0)88335 7862
> >
> > E
> >
> > address@hidden
> >
> >
> >
> >
> > [Logo]<http://www.deltares.com/>
> >
> > www.deltares.com<http://www.deltares.com/>
> >
> > Postbus 177
> > 2600 MH Delft
> >
> > [Deltares Twitter]
> > <http://www.twitter.com/deltares>
> >
> > [Deltares LinkedIn]<http://www.linkedin.com/company/217430>
> >
> > [Deltares
> > Facebook]<https://www.facebook.com/pages/Deltares/154189334634001>
> >
> >
> >
> > [Think before printing]Please consider the environment before printing
> > this email
> >
> >
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden]
> > Sent: vrijdag 2 maart 2018 16:26
> > To: Peter Herman
> > Subject: [netCDF #JZZ-916731]: problems in building netcdf 4.6.0
> >
> >
> > Peter Herman,
> >
> > Your Ticket has been received, and a Unidata staff member will review it 
> > and reply accordingly. Listed below are details of this new Ticket. Please 
> > make sure the Ticket ID remains in the Subject: line on all correspondence 
> > related to this Ticket.
> >
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> > Ticket ID: JZZ-916731
> > Subject: problems in building netcdf 4.6.0
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> >
> >
> >
> > The NetCDF libraries are developed at the Unidata Program Center, in 
> > Boulder, Colorado, funded primarily by the National Science Foundation.
> >
> > All support requests are handled by the development team. No dedicated 
> > support staff are funded at this time. For this reason we cannot guarantee 
> > response times, nor that we can resolve every support issue, although we do 
> > our best to respond within 72 hours.
> >
> > It is in the nature of support requests that the same question is asked 
> > many times. We urge you to search the support archives for material 
> > relating to your support request:
> >
> > http://www.unidata.ucar.edu/search.jsp?support&netcdf
> >
> > If you are having trouble building netCDF, please take a look at the 
> > "Building NetCDF" page:
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/building.html
> >
> > or the (unfortunately somewhat out-of-date) NetCDF Build Troubleshooter 
> > page:
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/troubleshoot.html
> >
> > Windows users should see the FAQ list:
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#windows_netc
> > df4_2
> >
> > Complete documentation (including a tutorial, and sample programs in C, 
> > Fortran, Java, and other programming languages) can be found on the netCDF 
> > Documentation page:
> >
> > http://www.unidata.ucar.edu/software/netcdf/docs/
> > http://www.unidata.ucar.edu/software/netcdf/examples/programs/
> >
> > If you resolve your issue through one of these methods, please send a reply 
> > to this email, letting us know that you no longer need support. This will 
> > help us spend more time on netCDF development.
> >
> > Best regards,
> >
> > Unidata User Support
> >
> > DISCLAIMER: This message is intended exclusively for the addressee(s) and 
> > may contain confidential and privileged information. If you are not the 
> > intended recipient please notify the sender immediately and destroy this 
> > message. Unauthorized use, disclosure or copying of this message is 
> > strictly prohibited. The foundation 'Stichting Deltares', which has its 
> > seat at Delft, The Netherlands, Commercial Registration Number 41146461, is 
> > not liable in any way whatsoever for consequences and/or damages resulting 
> > from the improper, incomplete and untimely dispatch, receipt and/or content 
> > of this e-mail.
> >
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: JZZ-916731
> Department: Support netCDF
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> DISCLAIMER: This message is intended exclusively for the addressee(s) and may 
> contain confidential and privileged information. If you are not the intended 
> recipient please notify the sender immediately and destroy this message. 
> Unauthorized use, disclosure or copying of this message is strictly 
> prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, 
> The Netherlands, Commercial Registration Number 41146461, is not liable in 
> any way whatsoever for consequences and/or damages resulting from the 
> improper, incomplete and untimely dispatch, receipt and/or content of this 
> e-mail.
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: JZZ-916731
Department: Support netCDF
Priority: Normal
Status: Open
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.