Hi Dana
My original thread title
"Make the CMake Windows build static"
is a bit misleading, sorry.
HDF5, in the NCO and netCDF cases, is only one of several input libraries.
You can think of both HDF5 and netCDF in 2 ways
1) A library that will be used as *input* for a program (EXE file) that any
HDF/netCDF user wants to build, his/her own program.
In this case he will need your LIB (in case he wants to compile) or the DLL (if
he just wants to run the EXE)
2) A final program destination already done (EXE) , like h5dump or ncdump, that
can be either statically built (no DLL) or dynamically built (need DLL)
For NCO, only case 2) exists. NCO is just the final program, not an input
library to anyone.
We only distribute EXE files, not LIB files, like you do.
If I am taking the HDF5 binaries you provide, and combine with other several
other library binaries, and even other libraries source code ,
in a combination of several LIB, DLL, and EXE files, dynamic is the way to go,
because of the reasons you mention.
Note: just the binaries. Not the whatever way I have to obtain them, in this
case, a Cmake compiler flag.
But, on the other hand, if I take all the source code only from these
“libraries”, and combine then into an EXE file statically build,
then I can consider all those “libraries” not libraries any more, just pieces
of source files that will produce an EXE.
One note, though, and here I would like you to correct me, if I’m wrong.
I can separate all these individual “libraries”, like HDF5, into an
intermediate input file, a LIB file,
that I use *only* for my compilation into an EXE, *not* to distribute to anyone.
Take the simplest case of 2 source files
a.c that I compile into a.LIB
main.c that I compile into main.EXE with a link input of a.LIB
If I compile both of these Visual Studio projects with the same static flag,
then,
there is no DLL dependency, and I think the CRT issues you mention do not
happen.
Do you agree with this?
When I started building Windows software in the 90’s with the Visual Studio
from that time I was using the DLL option.
Why? It seemed like a good idea. Dozens of programs done, thousands of
compiler builds, why not just make them share the system DLL?
Many of this software I kept both the source and the final EXE, stored in a
backup drive. But I forgot to store also the DLL, just because I never cared
about them,
I was not distributing software to anyone, just doing my college things.
Fast forward 20 years, I want to run the EXE from that program. what happens?
DLL whatever year 1993 is missing, oh my , I forgot to backup it.
And now where can I find it?, nowhere :-)
The funny thing is that the EXE is still *binary* compatible regarding the
Windows year 1993 version and Windows 7 today.
I just cannot run it because I don’t have the DLL. If I *had* used the static
option, now, today, I could run that EXE.
So, that’s all what I was trying to say with my comments, “people, don’t use
DLLs, they are bad”
Totally different case as compared to distributing HDF5 as a library.
CMake is only an automatic generator of Visual Studio projects.
Like it is now, I can go to your generated projects and change the compiler
flags.
But I thought it would be a valuable feature for the community to have this
choice already built directly into the Cmake step, like Ward did for netCDF,
and John also did, I think, in the original setup, but you took off the final
distribution.
Is that right, John?
Because I think the option –enable-static/shared, is just a given thing with
the system, just a matter of adding a one liner to the script ?
So, regarding your comment
“Personally, I'm not keen on making it easy for people to statically link to
the CRT since it's a documented bad idea”
I don’t think you should be limiting our choices.
I am just asking to let us do your own choices in the way we compile our
software.
Pedro
------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/
----- Original Message -----
From: Dana Robinson
To: HDF Users Discussion List
Cc: netcdfgroup@xxxxxxxxxxxxxxxx
Sent: Sunday, June 09, 2013 5:52 PM
Subject: Re: [Hdf-forum] [netcdfgroup] Make the Cmake Windows build
staticplease !
Hi all,
Just thought I'd toss in my two cents on this:
Statically linking to other libraries when you are building your own library
is not considered a very good idea on Windows. The reason for this is that
Microsoft implements the C runtime (CRT) functionality on a per-library basis.
i.e., each version of the CRT (debug vs. release, 10.0 vs. 9.0, etc.) has its
own private version of the heap, file handles, and other CRT objects. This
state is absolutely not shared among the dlls. This means that you have to be
VERY careful to ensure that CRT objects are not manipulated by the wrong
library (e.g., calling 'free' on a buffer allocated in a different version of
the CRT). Not paying careful attention to this will result in subtle,
difficult to find bugs down the road. When you statically link to a version of
the CRT, you aren't fixing the problem, you are actually multiplying it. Each
static link gives you a separate CRT state, just as if you mixed dll versions.
In general, we try to be very good about this in the HDF5 library and it's
not really an issue in normal use, though I'm pretty sure our help desk still
fields questions that are resolved by ensuring all libraries and the executable
are using the same CRT.
Even if it's not that big of a problem for us, the official word from
Microsoft is:
"Because a DLL built by linking to a static CRT will have its own CRT state,
it is not recommended to link statically to the CRT in a DLL unless the
consequences of this are specifically desired and understood."
http://msdn.microsoft.com/en-us/library/abx4dbyh(VS.80).aspx
Yeah, the static link makes it super easy to build your software, but at the
risk of corrupt applications. Personally, I'd rather solve the "where are my
dlls" problem instead of the "why is my software mysteriously crashing or
corrupting data" problem. The first one is a LOT more tractable.
If anyone is sitting on a pile of money they need to spend, this might
actually be a fixable problem. The Win32 libraries do not have this linkage
issue, so if we extended our platform-independence layer so that our C calls
mapped to Win32 calls, the problem should go away, with almost no change to the
bulk of the source code.
Personally, I'm not keen on making it easy for people to statically link to
the CRT since it's a documented bad idea. If we're going to spend any money on
Windows library issues, I'd rather see us properly distribute debug libraries.
Cheers,
Dana
On Sat, Jun 8, 2013 at 3:48 PM, Pedro Vicente <pvicente@xxxxxxx> wrote:
Hi Elena, John, Ward, Allen, Michael
> Support for /MT(d) was removed from CMake because we do not have
resources to support it with the HDF software (including HDF5).
> If this is a desired feature, we will need someone to test it on a
regular basis (daily with development and release HDF5 branches).
> I had set up 4 nightly builds of hdf5, static and dynamic, with and
> without fortran, and unfortunately, all goes wrong when you start
> using ifort on windows because of the way the supplied fortran libs
> are linked (once you link the intel fortran libraries your link errors
> multiply and you have no control of those libraries so having an
> option to change static/dynamic is no help!)
> I would concur that
> 1) Adding a version stamp to the dll/lib names is desirable (and
easy to do)
> 2) Adding a simple static/dynamic run time link option (and easy to
do)
OK, so, why not just add now the simplest configurations that are already
working, and that Ward has already done for netCDF?
The only minimal configuration that I think is certainly needed for
everyone is item 2), the static/dynamic run time link option, the equivalent of
GNU automake --enable-static/shared
One of the things that I am interested in is to build a Cmake build system
for NCO also, so what do you all think of a common CMake system for HDF, netCDF
and NCO?
I mean, things like
1) common syntax options like netCDF's
-DBUILD_SHARED_LIBS=OFF
2) use of environment variables, with the same name for libraries and
header files names;
for example, I use on my projects for the location of the curl library, the
name
LIB_CURL
and netCDF uses
CURL_LIBRARY
this would make things simpler for anyone that needs to build HDF, netCDF
and NCO.
Elena,
if you need to discuss in detail, call me (same number still), or we can
set Skype video/audio joint conference calls if needed.
Pedro
------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/
----- Original Message -----
From: Biddiscombe, John A.
To: HDF Users Discussion List
Cc: help@xxxxxxxxxxxx ; netcdfgroup@xxxxxxxxxxxxxxxx
Sent: Friday, June 07, 2013 1:05 AM
Subject: Re: [netcdfgroup] [Hdf-forum] Make the Cmake Windows build
staticplease !
I’d just like to second Elena’s comment below. As one of the people
responsible for the cmake implementation of hdf5, I had a really hard time with
this static/dynamic CRT link issue.
I had set up 4 nightly builds of hdf5, static and dynamic, with and
without fortran, and unfortunately, all goes wrong when you start using ifort
on windows because of the way the supplied fortran libs are linked (once you
link the intel fortran libraries your link errors multiply and you have no
control of those libraries so having an option to change static/dynamic is no
help!)
So there was not a simple combination that was easy for all setups and we
basically opted to remove most of the options to get a combination that worked
- It took up so much of my time that I didn’t ‘finish the job cleanly’ and
allow the user to decide for all cases.
I would concur that
1) Adding a version stamp to the dll/lib names is desirable (and
easy to do)
2) Adding a simple static/dynamic run time link option (and easy to
do)
I’m happy to test patches if people create them, but all my stuff works
fine, so I’m not volunteering!
JB
[I would also personally favour dynamic linking in 99% of cases. When you
have plugin based architectures where objects are created and destroyed in
different modules, static linking is a no go. I do not suffer from DLL hell and
actually find it much much much easier to distribute binaries to people on
windows than linux]
--
John Biddiscombe, email:biddisco @.at.@ cscs.ch
http://www.cscs.ch/
CSCS, Swiss National Supercomputing Centre | Tel: +41 (91) 610.82.07
Via Trevano 131, 6900 Lugano, Switzerland | Fax: +41 (91) 610.82.82
From: Hdf-forum [mailto:hdf-forum-bounces@xxxxxxxxxxxxxxxxxx] On Behalf
Of Elena Pourmal
Sent: 06 June 2013 17:01
To: HDF Users Discussion List
Cc: help@xxxxxxxxxxxx; netcdfgroup@xxxxxxxxxxxxxxxx
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !
All,
Support for /MT(d) was removed from CMake because we do not have
resources to support it with the HDF software (including HDF5).
If this is a desired feature, we will need someone to test it on a
regular basis (daily with development and release HDF5 branches). If there are
volunteers, please contact Allen. We will go from there.
Thank you!
Elena
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Elena Pourmal
Director of Technical Services and Operations
The HDF Group
1800 So. Oak St., Suite 203,
Champaign, IL 61820
www.hdfgroup.org
(217)531-6112 (office)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
On Jun 6, 2013, at 3:42 AM, Pedro Vicente wrote:
Hey , Allen & Ward, next time I send an email to *both* of your group
lists, can you just do a "reply-all" ? :-) ... otherwise , we get half of the
answers.
>>Building NetCDF statically is already an option, by passing
-DBUILD_SHARED_LIBS=OFF during configuration.
>>This will build the netcdf libraries and utilities statically, avoiding
the direct dependency on MSVCR100.dll.
>>They will, however, still inherit any downstream .dll dependencies from
the curl, hdf and zlib libraries.
Exactly, Ward, netCDF inherits HDF, that 's why I am bugging Allen here
:-)
And, yes, the solution is to build *all* downstream code in the *same*
way, either all static or all dynamic;
mixing both will get you linking errors.
>>> You might be able to work around this by downloading (or building)
static versions of these libraries.
Yes, Ward, this can be done by editing the HDF Cmake generated VS
projects, and changing all of them to static, which I just did.
In the curl case, you can do this by editing the file MakefileBuild.vc
and change all /MD (dynamic) to /MT (static)
Runtime library configuration
!IF "$(RTLIBCFG)"=="static"
#RTLIB = /MT
RTLIB = /MTd
RTLIB_DEBUG = /MTd
!ELSE
#RTLIB = /MD
#RTLIB_DEBUG = /MDd
>>Our binaries redistribute the MS CRT dlls that are used to build the
binaries.
Hey, Allen, I was not asking for the *binaries* !
I was asking for an *option* in your CMake system that allows
anyone who wants to generate the Visual Studio projects have them
*already* "static ready" if they wish...
So that, next time I need to generate everything from scratch I don't
have to go each to each one of them (50 + projects ) and click the static
button 50 times ...
>>Because of the danger of using different CRTs (HDF lib uses one and
user
>>program uses different one) and the possible memory corruption with
>>allocations, we build with /MD and provide the CRT with our binary.
I don't pretend to know everything, but when I don't (often :-) ) I search
read this
http://stackoverflow.com/questions/14749662/microsoft-visual-studio-c-c-runtime-library-static-dynamic-linking
quote
" i personally prefer statically linked. i hate scrambling around looking
for the right redist/dll/etc."
that's what I do, I set to all static , and then move on to do more
interesting things, like writing software, than dealing with DLL idiosyncracies.
quote
"Using /MT is risky if you create DLLs as well as an EXE. You'll end up
with multiple copies of the CRT in your program.
This was especially a problem with earlier versions of VS where each CRT
would get its own heap, "
Was this the problem you meant ?
This might be true if you are distributing *binaries* with both the EXE
and DLL. Or if you are linking your code against a 3rd party library *without*
the code,
someone gave you a LIB and a DLL only.
In the HDF, netCDF and NCO worlds none of this is true: all sources are
available , no secrets here :-)
And you are lucky, Allen , because you only have the downstream ZLIB, and
netCDF only has curl
Here's a list of all NCO dependencies
zlib ,
HDF5,
curl,
netCDF, including OpenDAP, thank you for that :-)
ANTLR, a parser generator for ncap2
GSL, the GNU scientific library
UDUnits, Unidata units (Not yet available for Windows)
Regular Expressions (Not yet available for Windows, tough one this one )
I have Visual Studio projects for all these (except UDUnits and Regular
Expressions)
, because I need to build the *source* , as you can see many projects to
change to static/dynamic/32/64bit/debug/release combinations :-)
Pedro
------
Pedro Vicente, Earth System Science
University of California, Irvine
http://www.ess.uci.edu/
----- Original Message -----
From: "Allen Byrne" <byrn@xxxxxxxxxxxx>
To: <hdf-forum@xxxxxxxxxxxxxxxxxx>
Sent: Wednesday, June 05, 2013 6:18 AM
Subject: Re: [Hdf-forum] Make the Cmake Windows build static please !
> Our binaries redistribute the MS CRT dlls that are used to build the
binaries.
> Because of the danger of using different CRTs (HDF lib uses one and
user
> program uses different one) and the possible memory corruption with
> allocations, we build with /MD and provide the CRT with our binary.
>
> Hopefully, as more people use CMake and build a common knowledge of
using
> CMake, those wishing to build alternative versions of HDF will share
their
> code changes.
>
> Allen
>
> On Wednesday, June 05, 2013 09:34:37 AM Michael Jackson wrote:
>> Funny, I actually _prefer_ the DLL builds because I distribute the
runtime
>> C/C++ libraries as allowed by MS. Depending on how one is using the
HDF5
>> executables/libraries having each library linked statically against
their
>> own C/C++ libraries can also lead to problems because of how memory is
>> allocated/deallocated in each library version. There are 2 evils here
and
>> the idea is to pick the least of them. If anything I would petition the
>> HDFGroup to provide BOTH a dynamically linked AND a statically linked
>> runtime version.
>>
>> Just my 2 cents. Then again, I build my own HDF5 for our project
precisely
>> because of issues like this.
>> ___________________________________________________________
>> Mike Jackson Principal Software Engineer
>> BlueQuartz Software Dayton, Ohio
>> mike.jackson@xxxxxxxxxxxxxx www.bluequartz.net
>>
>> On Jun 5, 2013, at 8:24 AM, Pedro Vicente <pvicente@xxxxxxx> wrote:
>> > Hi Allen, Ward
>> >
>> > I have a request regarding your new CMake Windows build system,
could you
>> > add an option to make the build static regarding the Microsoft
libraries
>> > (MSVCR100D.dll) ?
>> >
>> > Starting with version 4.3.1, NCO
>> >
>> > http://nco.sourceforge.net/
>> >
>> > uses the HDF5 and netCDF Windows libraries made with your CMake
system,
>> > and this is causing problems for NCO users, as explained here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9830/topic/8345151
>> >
>> > and here
>> >
>> > https://sourceforge.net/projects/nco/forums/forum/9829/topic/8417103
>> >
>> >
>> > This is just a matter of changing the compiler flag to /MT(d)
>> >
>> > http://msdn.microsoft.com/en-us/library/2kzt1wy3.aspx
>> >
>> > Using a dynamic build is just a bad idea, because of these DLL
issues.
>> >
>> > I have some Windows executables from code I did in the early 90's ,
that
>> > unfortunately I cannot run today, just because I linked them with
DLLs,
>> > with the DLLs from the Visual Studio from that time, that do not
exist
>> > anymore (it seems every new version they change the Visual Studio
Dlls).
>> >
>> > Because of this I do not use Dlls, I know that eventually something
will
>> > go wrong :-)
>> >
>> > Pedro
>> >
>> > ------
>> > Pedro Vicente, Earth System Science
>> > University of California, Irvine
>> > http://www.ess.uci.edu/
>> >
>> >
>> > _______________________________________________
>> > Hdf-forum is for HDF software users discussion.
>> > Hdf-forum@xxxxxxxxxxxxxxxxxx
>> >
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>>
>> _______________________________________________
>> Hdf-forum is for HDF software users discussion.
>> Hdf-forum@xxxxxxxxxxxxxxxxxx
>> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
> Hdf-forum is for HDF software users discussion.
> Hdf-forum@xxxxxxxxxxxxxxxxxx
> http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
>
> _______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@xxxxxxxxxxxxxxxxxx
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
--------------------------------------------------------------------------
_______________________________________________
netcdfgroup mailing list
netcdfgroup@xxxxxxxxxxxxxxxx
For list information or to unsubscribe, visit:
http://www.unidata.ucar.edu/mailing_lists/
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@xxxxxxxxxxxxxxxxxx
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
------------------------------------------------------------------------------
_______________________________________________
Hdf-forum is for HDF software users discussion.
Hdf-forum@xxxxxxxxxxxxxxxxxx
http://lists.hdfgroup.org/mailman/listinfo/hdf-forum_lists.hdfgroup.org
