[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[netCDF #JZZ-916731]: problems in building netcdf 4.6.0

I am not a lawyer :-), but my understanding is that code is not
open source unless you explicitly say so. If you do not want to
make a github pull request, then the simplest solution is probably
to just add a note here that the patches are released under,
say, the Apache 2 license or the BSD 3-clause license.

I remember about the e.g. sscanf problem. I think that there are
a number of functions where vis.studio  appends an underscore.
We have a specific include file -- include/ncconfigure.h --
in which to put macros to get the names right. I suspect that
many of your changes can go into that.

I think the case you are ignoring is Cygwin. I think it is the case
_WIN32 is defined under Cygwin, but Cygwin mostly conforms to linux+gcc.

So, I still prefer using _MSC_VER rather than e.g. _WIN32 because
almost all of the problems we have with this occur when using the
visual studio compiler. Using macro redefs like #define sscanf sccanf
takes care of most of the rest. In the few cases where MINGW is an
issue, I prefer to deal with that case specifically. And not using _WIN32
avoids the cygwin issue.


The TEMP case is as you say os dependent. I think its use is now
restricted to libdispatch/dutil.c


> >There is a problem. I do not believe we can accept patches because of 
> >potential copyright issues. Either you need to explicitly renounce all 
> >copyright on the patches or (better) create a pull request on github 
> >containing the proposed changes.
> 
> 
> What a strange problem. Of course I do not claim copyright, this is open 
> source and based on sharing work, isn't it? Anyway, I will try and make a 
> pull request. I have never done that before, but I will have a look how to do 
> it
> 
> > Is there any relatively simple characterization of the differences between 
> > vis.studio and mingw with respect to, say, headers?
> >There is, as a recall one place where we do make the distinction because 
> >legal files paths for mingw can differ from those accepted by vis.studio. I 
> >think it is, as a rule, best to include/exclude a header by detecting if it 
> >is present using the
> > configure.ac macros and having a flag in config.h. However, I admit to not 
> > always following this rule.
> 
> 
> With respect to headers, there are a number of flags that depend on the 
> availability of certain functions. I have made use of that where possible, 
> e.g. for the strlcat function. This fits perfectly into the existing code in 
> this respect. However, apart from that, there are a number of places where a 
> MINGW build is dependent on the Microsoft runtime library. This library is 
> strangely non-standard in some respects. In particular, the sscanf function 
> is different from the implementation in gcc. sscanf in the Microsoft runtime 
> does not recognize NaN, and further it cannot use %hhd specifier. Further, 
> the definition of the 'TEMP' environmental variable is OS-specific and 
> differs between Windows and Unix. Creating a /tmp directory fails in Windows. 
> In all these places, where the already existing patches for _MSC_VER are 
> actually not dependent on the compiler used, but on the operating system (and 
> runtime library), I have now consequently replaced the #ifdef _MSC_VER with 
> #ifdef _
 WIN32. I did it that way because I assumed the _WIN32 flag will also be set 
when one uses the Microsoft C compiler, but I cannot check that because I do 
not have the compiler. If it does not work that way, my patches will not be 
valid, or should be completed with a statement that sets _WIN32 whenever 
_MSC_VER is set.
> 
> >Unfortunately, I am not in a position to search the internals of MatLab to 
> >see how it uses the DOD {} construct. So, I guess this is is going to be a 
> >case-by-case set of fixes over time. In the mean-time, you can force a 
> >specific strlen by suffixing your url with '#stringlength=nnn' or (for a 
> >specific variable) '#stringlength_var=nnn' where 'var' is the fully 
> >qualified name of the variable
> 
> Thanks for the tip. I can't look inside MatLab either, so yes it looks like 
> it will be a heuristic process. But chances are it is not too complicated, as 
> it only seems to concern string variables and there is a clear syntax 
> followed for declaring the strlen in the DAS file.
> 
> 
> 
> Peter
> 
> 
> 
> 
> 
> 
> > Thanks for your reply. With respect to the patches: I do not want to hurry 
> > you up, just wanted to inform you about what I have been doing, hoping that 
> > it will also be useful for other developments. I am now making an inventory 
> > of all places in the software where the switch _MSC_VER is used, checking 
> > whether the switch is used for a compiler-dependent or rather operating 
> > system dependent choice. In the latter case, it would be better to replace 
> > it by _WIN32, which will also cover Microsoft C compiler but is 
> > generalizable to MINGW MSYS2. If I find the need for any other patches, I 
> > will let you know.
> >
> >
> >
> > With respect to the unreadable file: I am not a specialist at all on DODS, 
> > and I cannot help you with a clear description of the protocol. In fact, it 
> > is my impression that the use of the data attributes (in the DAS output) to 
> > store string length is not entirely orthodox with respect to the DODS 
> > protocol. It says (on opendap.org) that the DAS file is used to transmit 
> > data attributes that can be anything but size and structure of the 
> > variables, which should go in the DDS. However, the DDS apparently has no 
> > place holder for string length of individual variables, so it looks as if 
> > libraries (such as the Matlab library that was used to produce this 
> > particular problem file) abuse DAS to store information on string length of 
> > individual variables that they cannot get rid of in DDS.
> >
> > My advice would be to have a look at how exactly Matlab does this, as 
> > Matlab is probably the dominant library used to produce this type of files. 
> > Or to use Matlab to produce a variety of openDAP files, and then check 
> > whether they are readable. In the end, apart from the standards, it may be 
> > best to take a pragmatic view. If this is how people do it in actual files 
> > stored on openDAP servers, it is better to make sure one can read it...
> >
> >
> >
> > Best wishes,
> >
> > Peter
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > Peter Herman
> >
> > T
> >
> > +31(0)88335 7862
> >
> > E
> >
> > address@hidden
> >
> >
> >
> >
> > [Logo]<http://www.deltares.com/>
> >
> > www.deltares.com<http://www.deltares.com/>
> >
> > Postbus 177
> > 2600 MH Delft
> >
> > [Deltares Twitter]
> > <http://www.twitter.com/deltares>
> >
> > [Deltares LinkedIn]<http://www.linkedin.com/company/217430>
> >
> > [Deltares
> > Facebook]<https://www.facebook.com/pages/Deltares/154189334634001>
> >
> >
> >
> > [Think before printing]Please consider the environment before printing
> > this email
> >
> >
> > -----Original Message-----
> > From: Unidata netCDF Support [mailto:address@hidden]
> > Sent: maandag 12 maart 2018 22:14
> > To: Peter Herman
> > Cc: address@hidden; Peter Herman
> > Subject: [netCDF #JZZ-916731]: problems in building netcdf 4.6.0
> >
> > Thank you for the patches. It may take us a while to work thru them.
> > You say:
> > > Although these dimensions are given in the dds, the netcdf opendap
> > > software assumes default string length of 64 and neglects the given
> > > string lengths.
> >
> > The problem is that I could never find a defined standard about things like 
> > DODS attribute groups and where they can occur.
> > The netcdf-c code will translate and use DODS { Int32 strlen 211; } if it 
> > is at the top level of the .das file but (currently) not anywhere else.
> > When the code was written (a decade ago?) and if my memory is correct, the 
> > only examples I could find had only a global DODS attribute group.
> >
> > If you can point me to such a standard describing DODS (and other groups 
> > such as NC_GLOBAL) I would be happy to implement it.
> > In the meantime, your usage seems reasonable, so I will add a github issue 
> > to implement the DODS {} attribute group on individual variables, but I 
> > cannot promise when I will get to it.
> >
> > >
> > > In previous emails I explained that I am trying to build netcdf with 
> > > opendap capabilities in MINGW/MSYS2, because I want to use the library in 
> > > a package of the data analysis software R. I had several problems that I 
> > > solved sequentially in the following way:
> > >
> > >
> > >
> > > 1.     Some adaptations to CMake were needed - patch 1, copied from the 
> > > MSYS2 system, but addition of an extra measure (make mkstemp unavailable, 
> > > because it generated problems in compiling the R package)
> > >
> > > 2.     Typing error correction in dutil.c
> > >
> > > 3.     Export strlcat to the netcdf.dll, so that it is available for 
> > > subroutines that normally cannot use it because in the standard set-up it 
> > > is not included into the dll
> > >
> > > 4.     Change ddispatch.c such that it uses the Windows TEMP 
> > > environmental variable to make a temp file
> > >
> > >
> > >
> > > I explained these different patches in my previous e-mails. However, I 
> > > still had problems reading certain opendap files. The reason for this was 
> > > that these files used NaN as fill_value, and the netcdf routine dapcvt 
> > > could not handle this exception. In order to detect this, I had to be 
> > > able to switch on the log/debug mode of opendap, which I did in patch no. 
> > > 5. I subsequently patched the routine dapcvt.c so that it no longer 
> > > chokes on NaN. However, I feel my solution is not very elegant and I am 
> > > not sure that it will always be consistent. I include the patches also in 
> > > this file.
> > >
> > >
> > >
> > > 5.     (a non-patch) I use this to switch debug logging on or off
> > >
> > > 6.     A patch of dapcvt.c so that it converts strings to numbers 
> > > correctly when using the Windows run-time libraries (which sscanf does by 
> > > default), and so that it is resistant to NaN entries. The patch works for 
> > > NaN in double fields, but is not tested on other cases.
> > >
> > >
> > >
> > > Question: I remarked that elsewhere in the netcdf source files there are 
> > > routines that properly handle NaN, Inf etc., as well as convert strings 
> > > to certain types (e.g. long integers). I wondered whether these routines 
> > > could be used here too? I have the impression that this could be more 
> > > resilient than the solution presented by the present patch, and may not 
> > > require much additional programming. I would be interested in your 
> > > response to this suggestion.
> > >
> > >
> > >
> > > As for now, I am able to read all openDAP files that were on my
> > > list, with one exception. The file
> > > http://opendap.deltares.nl/thredds/dodsC/opendap/rijkswaterstaat/wat
> > > er base/catalog.nc does not read in correctly. However, this is not
> > > related to my build, as it also occurs when I use the ncdump build
> > > under Linux (Ubuntu). Maybe this file is badly constructed, or there
> > > is a bug in the opendap software that makes it impossible to read.
> > > Note that the netcdf file that was used by the openDAP server is
> > > readable, however. The problem is in the translation of the
> > > dimensions of the character arrays. Although these dimensions are
> > > given in the dds, the netcdf opendap software assumes default string
> > > length of 64 and neglects the given string lengths. I do not know
> > > why this is. I include the files ncdumpubuntu.txt for the output of
> > > using ncdump with the openDAP file on Ubuntu, ncdumpWindows.txt for
> > > the same operation with my build of ncdump on Windows, and
> > > ncdumpcatalognc.txt for the ou
> > tput of using ncdump (on Windows) directly on the netcdf file, so not 
> > through openDAP. This contains the 'true' structure of the file.
> > >
> > >
> > >
> > > I am looking forward to your reaction
> > >
> > >
> > >
> > >
> > >
> > > With kind regards,
> > >
> > > Peter
> > >
> > >
> > >
> > >
> > > Peter Herman
> > >
> > > T
> > >
> > > +31(0)88335 7862
> > >
> > > E
> > >
> > > address@hidden
> > >
> > >
> > >
> > >
> > > [Logo]<http://www.deltares.com/>
> > >
> > > www.deltares.com<http://www.deltares.com/>
> > >
> > > Postbus 177
> > > 2600 MH Delft
> > >
> > > [Deltares Twitter]
> > > <http://www.twitter.com/deltares>
> > >
> > > [Deltares LinkedIn]<http://www.linkedin.com/company/217430>
> > >
> > > [Deltares
> > > Facebook]<https://www.facebook.com/pages/Deltares/154189334634001>
> > >
> > >
> > >
> > > [Think before printing]Please consider the environment before
> > > printing this email
> > >
> > >
> > > -----Original Message-----
> > > From: Unidata netCDF Support
> > > [mailto:address@hidden]
> > > Sent: vrijdag 2 maart 2018 16:26
> > > To: Peter Herman
> > > Subject: [netCDF #JZZ-916731]: problems in building netcdf 4.6.0
> > >
> > >
> > > Peter Herman,
> > >
> > > Your Ticket has been received, and a Unidata staff member will review it 
> > > and reply accordingly. Listed below are details of this new Ticket. 
> > > Please make sure the Ticket ID remains in the Subject: line on all 
> > > correspondence related to this Ticket.
> > >
> > > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > > Unidata inquiry tracking system and then made publicly available through 
> > > the web.  If you do not want to have your interactions made available in 
> > > this way, you must let us know in each email you send to us.
> > >
> > > Ticket ID: JZZ-916731
> > > Subject: problems in building netcdf 4.6.0
> > > Department: Support netCDF
> > > Priority: Normal
> > > Status: Open
> > >
> > >
> > >
> > > The NetCDF libraries are developed at the Unidata Program Center, in 
> > > Boulder, Colorado, funded primarily by the National Science Foundation.
> > >
> > > All support requests are handled by the development team. No dedicated 
> > > support staff are funded at this time. For this reason we cannot 
> > > guarantee response times, nor that we can resolve every support issue, 
> > > although we do our best to respond within 72 hours.
> > >
> > > It is in the nature of support requests that the same question is asked 
> > > many times. We urge you to search the support archives for material 
> > > relating to your support request:
> > >
> > > http://www.unidata.ucar.edu/search.jsp?support&netcdf
> > >
> > > If you are having trouble building netCDF, please take a look at the 
> > > "Building NetCDF" page:
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/building.html
> > >
> > > or the (unfortunately somewhat out-of-date) NetCDF Build Troubleshooter 
> > > page:
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/troubleshoot.html
> > >
> > > Windows users should see the FAQ list:
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/faq.html#windows_ne
> > > tc
> > > df4_2
> > >
> > > Complete documentation (including a tutorial, and sample programs in C, 
> > > Fortran, Java, and other programming languages) can be found on the 
> > > netCDF Documentation page:
> > >
> > > http://www.unidata.ucar.edu/software/netcdf/docs/
> > > http://www.unidata.ucar.edu/software/netcdf/examples/programs/
> > >
> > > If you resolve your issue through one of these methods, please send a 
> > > reply to this email, letting us know that you no longer need support. 
> > > This will help us spend more time on netCDF development.
> > >
> > > Best regards,
> > >
> > > Unidata User Support
> > >
> > > DISCLAIMER: This message is intended exclusively for the addressee(s) and 
> > > may contain confidential and privileged information. If you are not the 
> > > intended recipient please notify the sender immediately and destroy this 
> > > message. Unauthorized use, disclosure or copying of this message is 
> > > strictly prohibited. The foundation 'Stichting Deltares', which has its 
> > > seat at Delft, The Netherlands, Commercial Registration Number 41146461, 
> > > is not liable in any way whatsoever for consequences and/or damages 
> > > resulting from the improper, incomplete and untimely dispatch, receipt 
> > > and/or content of this e-mail.
> > >
> > >
> >
> > =Dennis Heimbigner
> > Unidata
> >
> >
> > Ticket Details
> > ===================
> > Ticket ID: JZZ-916731
> > Department: Support netCDF
> > Priority: Normal
> > Status: Open
> > ===================
> > NOTE: All email exchanges with Unidata User Support are recorded in the 
> > Unidata inquiry tracking system and then made publicly available through 
> > the web.  If you do not want to have your interactions made available in 
> > this way, you must let us know in each email you send to us.
> >
> >
> > DISCLAIMER: This message is intended exclusively for the addressee(s) and 
> > may contain confidential and privileged information. If you are not the 
> > intended recipient please notify the sender immediately and destroy this 
> > message. Unauthorized use, disclosure or copying of this message is 
> > strictly prohibited. The foundation 'Stichting Deltares', which has its 
> > seat at Delft, The Netherlands, Commercial Registration Number 41146461, is 
> > not liable in any way whatsoever for consequences and/or damages resulting 
> > from the improper, incomplete and untimely dispatch, receipt and/or content 
> > of this e-mail.
> >
> >
> 
> =Dennis Heimbigner
> Unidata
> 
> 
> Ticket Details
> ===================
> Ticket ID: JZZ-916731
> Department: Support netCDF
> Priority: Normal
> Status: Open
> ===================
> NOTE: All email exchanges with Unidata User Support are recorded in the 
> Unidata inquiry tracking system and then made publicly available through the 
> web.  If you do not want to have your interactions made available in this 
> way, you must let us know in each email you send to us.
> 
> 
> DISCLAIMER: This message is intended exclusively for the addressee(s) and may 
> contain confidential and privileged information. If you are not the intended 
> recipient please notify the sender immediately and destroy this message. 
> Unauthorized use, disclosure or copying of this message is strictly 
> prohibited. The foundation 'Stichting Deltares', which has its seat at Delft, 
> The Netherlands, Commercial Registration Number 41146461, is not liable in 
> any way whatsoever for consequences and/or damages resulting from the 
> improper, incomplete and untimely dispatch, receipt and/or content of this 
> e-mail.
> 
> 

=Dennis Heimbigner
  Unidata


Ticket Details
===================
Ticket ID: JZZ-916731
Department: Support netCDF
Priority: Normal
Status: Closed
===================
NOTE: All email exchanges with Unidata User Support are recorded in the Unidata 
inquiry tracking system and then made publicly available through the web.  If 
you do not want to have your interactions made available in this way, you must 
let us know in each email you send to us.