nix
nix copied to clipboard
NixOS
Port Nix to Meson
This pull request adds meson buildsystem support. meson allows for better cross platform support. and is a LOT easier to maintain when compared to autotools. There are also some modifications to fix compilation on FreeBSD.
TODO:
- add meson support for tests
- add meson support for docs
- add amazon s3 building
I'm not sure we're having all that much trouble maintaining the current autotools-based build system, and Nix's build setup is fairly focused around being able to bootstrap it with fairly minimal dependencies. Does meson generate scripts that end up in release tarballs that we can run without having meson available on the build host? I think this needs a lot of justification either way, since we'd be maintaining two build systems rather than one, because I doubt we can drop autotools anytime soon.
related:
https://github.com/NixOS/nix/issues/2503 https://github.com/NixOS/nix/pull/1439 https://github.com/NixOS/nix/pull/2613
I'm not sure we're having all that much trouble maintaining the current autotools-based build system, and Nix's build setup is fairly focused around being able to bootstrap it with fairly minimal dependencies. Does meson generate scripts that end up in release tarballs that we can run without having meson available on the build host? I think this needs a lot of justification either way, since we'd be maintaining two build systems rather than one, because I doubt we can drop autotools anytime soon.
the current autotools system leaves a lot to be desired for cross platform support. meson allows a more platform agnostic approach to the buildsystem. Moreover, this port is heavily commented to allow easy future support.
Ninja also compiles significantly faster than autotools/make. It would save developers a substantial amount of time and massively improve productivity.
this adds two dependencies to the source directory; meson & ninja (or muon & samuri). once configured, the build directory only requires ninja/samuri and the previous dependencies. it also removes the dependency on a linux userland
#2613
Related how? Is this based on @Ericson2314's work, or is this a from-scratch rewrite that just happens to share the same goal?
this adds two dependencies to the source directory; meson and ninja. once configured, the build directory only requires ninja and the previous dependancies.
I'm not sure how to parse this. To my knowledge, ninja doesn't really have conditionals, so if we wanted to produce source tarballs that don't depend on meson we'd have to build ones for specific targets, right?
#2613
Related how? Is this based on @Ericson2314's work, or is this a from-scratch rewrite that just happens to share the same goal?
This is a from-scratch write, but some parts are similar.
this adds two dependencies to the source directory; meson and ninja. once configured, the build directory only requires ninja and the previous dependancies.
I'm not sure how to parse this. To my knowledge, ninja doesn't really have conditionals, so if we wanted to produce source tarballs that don't depend on meson we'd have to build ones for specific targets, right?
im not quite sure what you're asking either :)
im not quite sure what you're asking either :)
Trying to tease out what the minimal dependencies for building from a source tarball would end up being, but perhaps that's material for #2503.
im not quite sure what you're asking either :)
Trying to tease out what the minimal dependencies for building from a source tarball would end up being, but perhaps that's material for #2503.
all dependencies should be labeled inside of the root meson.build (sans meson & ninja)
Yes, this does add python as a build time dep. But I think that's a fine price to pay to get rid of a Unix as a build time dep. (Also, I care more about run time than build time deps as one can always cross compiler from a more normal to a weirder platform.)
Yes, this does add python as a build time dep. But I think that's a fine price to pay to get rid of a Unix as a build time dep. (Also, I care more about run time than build time deps as one can always cross compiler from a more normal to a weirder platform.)
not to mention cross compilation on meson is considerably easier other platforms
The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (https://github.com/NixOS/nix/pull/2748) without first having to learn how to do that with a completely different build system.
Ninja also compiles significantly faster than autotools/make, to an almost embarrassing extent.
I'm a bit skeptical about this. Some numbers would be helpful. I mean, an incremental "make" in Nix takes 0.05s on my system to determine that it doesn't have to do anything. So the "make" overhead is fairly non-existent.
BTW we don't really have an "autotools-based build system" - we have a Make-base build system. We're not using automake or libtool (anymore). The only autotool we're still using is autoconf and I'd be happy to get rid of it (it doesn't do much that couldn't be done using some Make / C++ conditionals). The approach of detecting rather than specifying platform characteristics is really a remnant of the era when people had to support dozens of obscure Unix systems.
The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system. I think it is more helpful to the community to use a system that many people can understand very easily instead of keeping the same system because people are used to it. Meson is VERY simple to understand and has brilliant documentation.
I am also willing to help explain every single line of each script, what it does, why, if/where this is performed in the current build system and why it was included/added.
Ninja also compiles significantly faster than autotools/make, to an almost embarrassing extent.
I'm a bit skeptical about this. Some numbers would be helpful. I mean, an incremental "make" in Nix takes 0.05s on my system to determine that it doesn't have to do anything. So the "make" overhead is fairly non-existent.
I'll run some compilation tests for some numbers shortly ^^
BTW we don't really have an "autotools-based build system" - we have a Make-base build system. We're not using automake or libtool (anymore). The only autotool we're still using is autoconf and I'd be happy to get rid of it (it doesn't do much that couldn't be done using some Make / C++ conditionals).
i believe i said "Ninja also compiles significantly faster than autotools/make". autotools isnt what is causing the awful build performance, make is. switching to ninja takes no overhead on the part of the user, you just type ninja instead of make.
The approach of detecting rather than specifying platform characteristics is really a remnant of the era when people had to support dozens of obscure Unix systems.
The purpose of this re-write is to allow better cross platform support. Just because a make file works on linux, it does not mean it works on minix/*bsd/darwin. The build files for bsd especially have bit rotted to a point where you cannot even compile the source without the modifications included in this PR.
I personally believe Nix should, at least, build on all supported platforms. switching to this meson system will help prevent things breaking in the future. I've also implemented methods to allow a more platform agnostic approach to the build.
compilation times
meson build compile start: 1571742622 compile end: 1571742805 total time: 183 seconds
make build compile start 1571743308 compile end: 1571743716 total time: 408 seconds
@edolstra FWIW meson is is popular among Gnome devs who very interested and C -> Rust things, and so the specific example you mentioned is very much supported.
As the killer app for me is removing the unix assumptions without maintain multiple build systems, I'll try to revive https://github.com/volth/nix-windows/ with this during or shortly after NixCon. (I would commit to during and not after if Windows weren't such a pain!)
The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.
How can we make sure that Eelco would be comfortable with Meson? Is anyone willing to be on call for help and provide teaching?
My experience with hacking on Nix is that compilation is quite slow, having faster builds would be a very welcome change to me. Especially since I don't C++ well enough and require more iterations to get anything done :D
The current build system has one major advantage: I know how it works. So I can do things like add some Rust code (#2748) without first having to learn how to do that with a completely different build system.
How can we make sure that Eelco would be comfortable with Meson? Is anyone willing to be on call for help and provide teaching?
https://github.com/NixOS/nix/pull/3160#issuecomment-544910012
My experience with hacking on Nix is that compilation is quite slow, having faster builds would be a very welcome change to me. Especially since I don't C++ well enough and require more iterations to get anything done :D
same ^^
@ p01arst0rm Are those compilation times doing the same thing? Namely an -O3 build? And with the same level of parallelism (-j...)?
@ p01arst0rm Are those compilation times doing the same thing? Namely an
-O3build? And with the same level of parallelism (-j...)?
(performed on different box than the first set) -j2 -O3
make compile start: 1572002763 compile end: 1572003137 total time: 374
meson compile start: 1572003191 compile end: 1572003556 total time: 365
still faster with identical c flags, repeated to check and it was still faster, but Its less dramatic when you set up optimisation flags that way.
update: the make build didnt even finish, i checked the logs and it errored out
-j4 -O3
make 1572009306 1572009610 304
meson 1572009781 1572010079 298
seems like the difference only matters the more restricted resources are. still faster though.
I would welcome this change.
I think autoconf and in particular m4 is very difficult to contribute to; in my last PR to improve nix's autoconf (#2979, which fixed nix autoconf bugs #257, #506, #967) I spent 2 days to improve it.
In contrast, packages in nixpkgs that use meson to which I recently contributed, like gstreamer, were easier to work with.
I expect that supporting meson will make it way easier for people to contribute to nix and help maintain its build (and prevent that it stops building on e.g. Debian and Ubuntu as happened in the issues mentioned above).
On the idea of using plain make without any configurator:
I would be surprised if it worked well if we want to provide a good workflow for not-in-nix-but-from-source builds, e.g. for other distros. The i3 window manager did for a long time use a plain make file but eventually switched away from it.
Systems like meson find the tools you need (compiler and dependencies, whatever they are named in cross or non-cross cases) by sane means (e.g. pkg-config), allow you to cleanly and unambiguously declare and provide optional dependencies, and give good error messages when something isn't found.
any updates on this? dont really know if i should keep working on this if its not going to get merged
@p01arst0rm we made the case at NixCon a good deal. Is my PR ready for you to merge? Then I think we just need to fix some Darwin build failures.
@p01arst0rm we made the case at NixCon a good deal. Is my PR ready for you to merge? Then I think we just need to fix some Darwin build failures.
Theres some things with your PR that arent quite right that im waiting on ^^
@p01arst0rm I in part wrote https://github.com/NixOS/rfcs/pull/56 because I think reviewing PRs like this is needlessly difficult without it.
I'll keep on plugging away on the windows stuff in the mean time (https://github.com/NixOS/nixpkgs/pull/73195 was what I last did) and that should bolster the case for this.
Sorry this is not a faster process.
@p01arst0rm I in part wrote NixOS/rfcs#56 because I think reviewing PRs like this is needlessly difficult without it.
I'll keep on plugging away on the windows stuff in the mean time (NixOS/nixpkgs#73195 was what I last did) and that should bolster the case for this.
Sorry this is not a faster process.
no problem, I just don't want this to bit rot :)
Sorry this has taken so crazily long. We should soon have a meeting for https://github.com/NixOS/rfcs/pull/56 at which point I think we can get it over the finish line.
If you have any energy left for this, master as started using Rust, so that will be a slightly more difficult thing to merge in. (Though meson supports Rust so I don't expect too much trouble.)