meson
meson copied to clipboard
mesonbuild
Windows Library naming conventions
Describe the bug Meson build system currently labels Windows static libraries by default with .a extension in order to distinguish between static and import libs. I understand that Meson supports overriding this behavior with "name_suffix" param.
Shouldn't Meson follow Windows conventions by default? How does CMake deal with this issue? Another approach is to use different library names, e.g. lib_xyz.lib and lib_xyz_static.lib or to put the build artifacts in different folders such as static and shared.
Expected behavior Meson should follow windows convention and name static libs with .lib extension by default.
Just for reference: vcpkg is renaming all static libs to follow the native naming on windows because otherwise CMake and Autotools +pkg-config are completely broken. Since it breaks pkg-config it also breaks meson itself.
I think the explanation for the conscious choice why things are the way they are is pretty good, but while it solves the collision of import-vs-static libs, it introduces a new collision for artefacts with a mingw ABI vs. MSVC ABI, which is what people are running into, now that the CMake 3.25 has been released, which considers .a on MSVC due to meson's naming choice (complicated by the outrageous Strawberry perl stuff, and the fact that it's installed by default in GHA images).
But I was wondering why it has to be libfoo.a? I get that it's aesthetically appealing to have similarly named artefacts on all platforms, but do the benefits here really outweigh the costs?
How about .lib.a (or .b, or ...)? That way, CMake could provide a saner choice (not considering mingw-built libfoo.a artefacts in MSVC builds, but still consider libfoo.lib.a, i.e. when the artefact has come from meson), and meson could still disambiguate import libs from static libs.
CC @eli-schwartz, I saw you commented on reddit about this. There have been more comments about this on the CMake PR. I don't have a horse in this race, except that - all things considered - I'd prefer more projects to add support for meson (and the breakage caused by this isn't helping that goal particularly AFAICS).
Does anyone have a proposal for a naming convention that would fix this issue without causing breakage in other cases. IIRC Rust has a different naming scheme for their libs but I don't remember the details (and I looked at that ages ago, maybe it has changed since).
Does anyone have a proposal for a naming convention that would fix this issue without causing breakage in other cases.
See above:
How about
.lib.a(or.b, or ...)?
AFAICT, it should be enough choose a different file extension other than vanilla .a on windows.
Part of the reason for originally choosing libfoo.a is because GNU and MSVC can mix under some circumstances (of course not including the Strawberry Perl badness) and as mentioned in the FAQ, mingw can only find libraries via -lfoo if they follow one of two particular naming schemes.
I suppose it's not beyond reason for them to add a third (and maybe even popularize convention between meson, gcc, and cmake to fix this problem once and for all).
mingw can only find libraries via
-lfooif they follow one of two particular naming schemes.
Is building on mingw with meson a common usecase (much less mixing in artefacts produced by MSVC)? It seems to me that mingw tends to stay in its own distribution bubble much much more than a cross-platform build tool like meson ever could, so reusing meson-built artefacts in mingw seems to me to be the least widespread[^1] usecase to sacrifice here...
OTOH, with the speed at which things diffuse in mingw-land (at least from some limited experience), I wouldn't expect a third way to find libraries (even if released today) to be broadly usable before 3-5 years, and presumably this situation should not stay in limbo that long...
[^1]: citation needed
Is building on mingw with meson a common usecase
Yes, because that includes both cygwin and MSYS2 runtime layers. :)
(much less mixing in artefacts produced by MSVC)?
That's a better question. The answer to that is "I dunno".
It's definitely done at least sometimes. @nirbheek can speak about gstreamer usage, for example.
If I had to take a completely wild guess, I'd guess that people who do development with a mingw compiler are significantly more likely to mix the two than people who do development with a MSVC compiler, though!
...
Due to the first part, it's absolutely, unquestionably correct to continue using mingw naming when running the mingw compiler. :D Fortunately, that part is easy to if/else.
Due to the second part, it might be plausible to use .lib.a or whatever if the compiler is MSVC -- we already have a handy self.get_using_msvc() for that (we need to determine whether to use .dll.a or .lib for shared library import libs).
If we did do that:
- cmake can look for those MSVC-ABI libraries specifically.
- Meson would probably continue to look for both
libfoo.aandlibfoo.lib.aas the meson compiler environment isn't leaky and prone to auto-misdetecting Strawberry Perl, so if you're looking for libraries with both ABIs together, you intended to do it. - People using non-Meson build systems AND who want to use libraries from both camps, would likely have some fiddling to do, tough luck
(FD: CMake developer)
FWIW, I don't find the "but it makes -lfoo work" argument convincing at all. I'd much prefer seeing full paths instead of decay into -L and -l flags for link lines (and searching for libraries at configure time gives you full paths already, so why break them apart in the first place?). Having to debug when decomposition puts some unintended library in front what I actually wanted isn't fun.
That's unfortunately totally broken, because full paths hardcode one or the other of static vs. shared libraries.
I do understand that shipping static and shared isn't a use case supported by cmake, so cmake's approach is internally consistent, but... it doesn't work generically, it's a practical problem for Unix users, and it's technically wrong for Windows as well even if in practice windows users often only bother building whichever linkage model will be used by the next step in the application they are currently building without a system library framework.
(But that doesn't mean Windows users never want both!)
Decomposition can't ever find the wrong thing if you do manual searching instead of letting the compiler interpret -l, because it's well within your power to search only in the -L location and ignore unintended libraries in other locations.
That doesn't mean we can't do better! *-config.cmake is inherently unfixable, but pkg-config could be fixed to learn new tricks, such as using a libraries field with split support for specifying the name and static-path: ... + shared-path: ... of the libraries.
... We could even call it "CPS" or something. :p But that is overly verbose, has way too much focus on things people don't need, doesn't nicely handle common pkg-config use cases, like dataonly, and making yet another tool instead of version 2.0 of an existing tool implies making a third ecosystem.
(Freedesktop pkg-config is effectively dead. Everyone tends to use pkgconf instead, these days. The pkgconf developer is interested in incrementally improving .pc files and probably moving to something that has proper quoting, like json. But it's probably a good idea to have the new universal package configuration recording and parsing tool be backwards compatible with the existing state of the art.)
(Note also that you still want to be able to emit the flags as -lfoo via an option, because some people want that. e.g. I seem to recall bug reports about libtool breaking horribly otherwise. Yes, I mentioned libtool. I'm deeply sorry to do that to you. ☹️)
hardcode one or the other of static vs. shared libraries.
your approach is already broken so ..... (in exactly the same way on windows)
pkg-config has the cmd line switch --msvc-syntax but meson seems to have ignored that one.
shipping static and shared isn't a use case supported by cmake
that statement is wrong. I see a lot of upstream building both. The difference is that it requires explicit targets to do so instead of implying it.
*-config.cmake is inherently unfixable
You don't mean *-config.cmake. That should be the user generated part and not the CMake generated part.
libtool breaking
libtool already breaks if you try to mix static and shared libs in the lookup unless you explicitly pass lt_cv_deplibs_check_method=pass_all
You don't mean *-config.cmake. That should be the user generated part and not the CMake generated part.
No, I mean the ecosystem which that's an entry point to. The user generated part is broken, and the cmake generated part is also broken.
libtool already breaks if you try to mix static and shared libs in the lookup
No. I mean libtool was breaking when using all shared libs all the time everywhere, if the shared lib is specified via a path instead of -lfoo.
(Why, I could not tell you.)
because full paths hardcode one or the other of static vs. shared libraries.
Any specific link should? One already needs to change the link line when switching, so why is this not suitable?
I do understand that shipping static and shared isn't a use case supported by cmake
It's not something supported through a "config" change (i.e., release vs. debug), but projects do indeed do this.
Decomposition can't ever find the wrong thing if you do manual searching instead of letting the compiler interpret -l, because it's well within your power to search only in the -L location and ignore unintended libraries in other locations.
Decomposition can fail with:
/lib/libA.so/lib/libB.so/lib2/libB.so
where -L/lib -L/lib2 -lA -lB won't find /lib2/libB.so. I'm talking about the linker here. All we have is the command line to communicate with the linker (last I checked), so how is doing the search manually going to fix that?
*-config.cmake is inherently unfixable, but pkg-config could be fixed to learn new tricks, such as using a libraries field with split support for specifying the name and static-path: ... + shared-path: ... of the libraries.
Yeah, CMake's config files use syntax that is not trivial for anything else to understand. I don't think it's the long-term solution for interop myself either (though I'd be surprised if CMake didn't convert whatever it reads into that format for a long overlap period). As for pkg-config, I think there are things that will be hard-pressed to be specified there, namely combination and reduction rules for flags (e.g., deduping -I flags or knowing that rpath directories are joined with :). I think it could be done for a static list, but my longer-term idea is to support user-defined usage requirements (e.g., "add X to PYTHONPATH to use me", "I have a path for plugins of application Y, please use path Z", or "collect all values of property P and put them into a JSON list"). I don't think a key/value setup is suitable there and some richer structure is required in general.
What I do not want to see is continuation of the "flag soup" that we have today. Not all compilers share flag spellings or even support the same features. There are requirements that depend on usage (e.g., "link to libpython if you are an executable or a plugin not being loaded by something necessarily already aware of Python, otherwise tell the linker to ignore undefined symbols" is not something I think one can express in a .pc without doing bespoke key generation) and trying to spell all of those out is destined for failure IMO.
pkg-config has the cmd line switch
--msvc-syntaxbut meson seems to have ignored that one.
We didn't ignore it. We don't use it because it has (had?) a bug where it would blindly put the .lib suffix on incorrect parts of the Libs: line and because doesn't give us anything special anyway. We'd need to manually parse the list of args to get rid of things like -lm or linker args that pkg-config can't translate to MSVC syntax for us.
Decomposition can fail with:
/lib/libA.so /lib/libB.so /lib2/libB.so
where -L/lib -L/lib2 -lA -lB won't find /lib2/libB.so. I'm talking about the linker here. All we have is the command line to communicate with the linker (last I checked), so how is doing the search manually going to fix that?
This is actually exactly the case that Meson handles better than most other build systems. If you wanted to pull libA and libB from two different prefixes and they have pkgconfig files, then you'd get:
$ export PKG_CONFIG_PATH=/bar/lib:/foo/lib
$ pkg-config --libs A
-L/foo/lib -lA
$ pkg-config --libs B
-L/bar/lib -lB
If you don't do decomposition, you get a linker line like:
$ cc -o foo -L/foo/lib -lA -L/bar/lib -lB
Which will actually put -Lfoo/lib first and pick up /foo/lib/libB.so. Meson does manual searching for libs while parsing the pkgconfig output so the linker line it generates will be:
$ cc -o foo /foo/lib/libA.so /bar/lib/libB.so
And if you call dependency() with static: true then it will search for static libs instead of shared libs.
Like you say, we only have the command-line to communicate with the linker, so it is up to us to provide APIs for the user to communicate to us (the build system) what they want to do, so we can make sure that the linker does the right thing.
Now, why is it a benefit to have libraries that can be found with -L -l pairs? Because the rest of the world doesn't do all this work. Autotools, Rust, and even make-based custom build systems (like openssl, ffmpeg, etc) expect -L -l pairs to work.
For example, if a Rust crate wants to link to your library using a pkgconfig file (which is something gstreamer, gtk, glib, etc use for Rust bindings), then the build system will need to parse the pkgconfig output and pass it to rustc correctly: https://github.com/rust-lang/pkg-config-rs/blob/master/src/lib.rs#L88-L111
because GNU and MSVC can mix under some circumstances
GNU / MSVC is actually a false split. The actual split is:
- C++ / C ABI
- What CRT is being used?
If you want to link libA with libB, if the two are C libraries, all you need to check is what CRT they are using. If both are using MSVCRT or UCRT then you're fine. Else, you can still link them but you need to be careful about memory management: https://learn.microsoft.com/en-us/cpp/c-runtime-library/potential-errors-passing-crt-objects-across-dll-boundaries
If you want to link libA with libB, if the two are C++, they must be built with the same C++ std and ABI, which means they must be built with the same major compiler version. IIRC MSVC breaks C++ ABI across compiler versions, and each C++ std has a different ABI. Also both must be linked to the same CRT.
So, the actual fix that cmake (and meson!) should employ is to check whether the static library contains only C objects, and in that case just link them together if they use the same CRT. And if it contains C++ objects, also check whether the C++ ABI is compatible.
We didn't ignore it. We don't use it because it has (had?) a bug where it would blindly put the .lib suffix on incorrect parts of the Libs: line and because doesn't give us anything special anyway.
This comment was not about meson but outside meson. E.g. generating link lines from pc files for e.g. msbuild.
can't translate to MSVC syntax for us.
that shouldn't happen and means the pc file is incorrect.
@mathstuf @bradking,
(I cannot comment on the cmake MR because I cannot seem to find my gitlab.kitware.com password, and I'm not sure how I originally created that account but my current email address isn't getting recovery emails but also seems to say it's already in use. Sorry for discussing this in the wrong place.)
I wonder if maybe the correct long-term solution for CMake is to check for libfoo.a but only for prefix paths that were manually added by the user, not ones that were autodetected from $PATH? It seems like the problem essentially boils down to:
- users who manually specify that the development environment consists of such and such carefully crafted locations, presumably verified that those libraries are the right ABI
- assumption historically works for Meson
- things break when combined with adding automatically detected locations to the development environment
- it may be possible to analyze found libraries and check whether they are compatible, but it's probably quite a bit of work
I don't know if CMake is set up to make this split between explicit vs. implicit prefixes, but if it could be done then CMake would once again be able to detect Meson-built static libraries, but also autotools ones and, I believe, the ones cmake itself creates when told to use the mingw GCC compiler (and the UCRT as a default CRT).
What do you think?
things break when combined with adding automatically detected locations to the development environment
I was thinking along those lines too, and have opened CMake Issue 24216 to discuss dropping the PATH-derived search prefixes.
This also affects using Meson-built libraries from Rust, which hardcodes the MSVC ABI to expect .lib: https://github.com/rust-lang/rust/blob/2ceed0b6cb9e9866225d7cfcfcbb4a62db047163/compiler/rustc_target/src/spec/windows_msvc_base.rs#L15
I was not aware of this thread, and there is indeed something to do in Rust as well... https://github.com/rust-lang/rust/issues/114013.
FWIW, I find Rust's foo.dll.lib name for import lib makes sense. But I don't really understand implications of doing such change in Meson.
So rustc produces import libraries that are unusable by mingw GCC which is otherwise capable of linking together mingw+ucrt libraries and MSVC libraries?
It is not clear to me why this is an advantage...
I haven't followed the whole thread here, what's the TL;DR? When doing -lfoo what are the filenames GCC looks for? What are the names MSVC looks for? is there any intersection?
In the likely case there is no intersection, I think ultimately the decision needs to come from the user. Only they know how that lib will be used, so we probably need a global naming_convention: <gcc, msvc, auto> option? That way user can build something with GCC but tell Meson to produce foo.lib instead of libfoo.dll.a because they know it's going to be used by MSVC.
Sorry if I miss the point here, I'm new to this topic, trying to understand.
Part of the reason for originally choosing
libfoo.ais because GNU and MSVC can mix under some circumstances (of course not including the Strawberry Perl badness) and as mentioned in the FAQ, mingw can only find libraries via-lfooif they follow one of two particular naming schemes.
The FAQ: https://mesonbuild.com/FAQ.html#why-does-building-my-project-with-msvc-output-static-libraries-called-libfooa
Also try searching mesonbuild/build.py for "libfoo".
The FAQ: https://mesonbuild.com/FAQ.html#why-does-building-my-project-with-msvc-output-static-libraries-called-libfooa
Oh, point 6 there is the part I missed, thanks.
So rustc produces import libraries that are unusable by mingw GCC which is otherwise capable of linking together mingw+ucrt libraries and MSVC libraries?
You mean that with GCC -lfoo won't find foo.dll.lib? It has to be named foo.lib?
So rustc produces import libraries that are unusable by mingw GCC which is otherwise capable of linking together mingw+ucrt libraries and MSVC libraries?
I'm talking specifically about Microsoft's MSVC, haven't gotten round to testing GCC here.
MSVC cannot find Meson generated libraries. Not because they're wrong in format; rustc tells LINK.EXE that they have the .lib extension (as per the TargetOptions, and per MSVC's own standard), but Meson does that rename to avoid the name clash. This is not detected in any way by rustc, and cannot be handled by Meson except by going over every single dependency and setting the name_suffix appropriately.
@xclaesse @eli-schwartz Once more, my remarks so far only cover the MSVC compiler -- I've not tried to build gst-plugins-rs with MSYS2, so I cannot comment or offer any insights yet.
MSVC cannot find Meson generated libraries.
MSVC cannot find libraries, it doesn't implement the concept of searching.
rustc tells LINK.EXE that they have the .lib extension (as per the TargetOptions, and per MSVC's own standard),
rustc can tell LINK.EXE whatever rustc wishes to tell it -- and rustc will have to tell it that they have a *.a or *.dll.a extension, if rustc wishes to support interoperability between mingw libraries built against the UCRT and cl.exe libraries, something that is commonly done in ecosystems such as gstreamer that need to build against a variety of open source libraries some of which only have an autotools build system.
MSVC doesn't have a standard, period, so that shouldn't be a problem.
MSVC cannot find Meson generated libraries.
You can use the verbatim link modifier for externs if you're just wanting to link libraries from rust that don't follow the target's conventions. E.g.:
#[link(name = "mylib.a", modifiers = "+verbatim")
extern "C" {
fn bar();
}
MSVC doesn't have a standard, period, so that shouldn't be a problem.
Quoting from https://learn.microsoft.com/en-us/cpp/build/reference/dot-lib-files-as-linker-input?view=msvc-170:
LINK accepts COFF standard libraries and COFF import libraries, both of which usually have the extension .lib.
To the best of my experience, Meson is the only build system to actually do something regarding that name clash, but unfortunately it does not follow the line above.
You can use the verbatim link modifier for externs if you're just worries about linking libraries from rust that don't follow the target's conventions. E.g.:
This one is not viable, because a downstream consumer would need to patch every single dependency to work around this.