liburing
liburing copied to clipboard
axboe
add meson based buildsystem v2
add a meson based build system
This is based on and supersedes the meson PR #297.
I am willing to maintain the meson build system and provide help for all those not familiar with meson (@tp-m and @stalkerg are also willing to help). Anybody can reach out to me using [email protected].
Additionally, I am already following the io_uring mailing list and will also respond there.
@axboe Should I post this also to the io_uring mailing list?
TODOS
- [x] support the no Glibc feature provided by the current build system
- [x] test the meson build system with GitHub bot
- [x] ~~decide the minimum required meson version~~ I settled with meson
>=0.53
git request-pull output:
The following changes since commit 1842b2a74f4e914cb094019d0f339baeffa3023b:
examples/io_uring-udp: Use a proper cast for `(struct sockaddr *)` argument (2022-07-26 10:46:47 -0600)
are available in the Git repository at:
https://github.com/fischerling/liburing/ meson-liburing
for you to fetch changes up to e29fafb12c40fa5de37b16bb21872c4f2476075c:
meson: update available examples to liburing 2.3 (2022-07-27 17:19:15 +0200)
----------------------------------------------------------------
Florian Fischer (8):
meson: update meson build files for liburing 2.3
meson: update available tests to liburing 2.3
meson: update installed manpages to liburing 2.3
meson: add default test setup running each test once
meson: support building without libc
meson: add 'raw' test suite
github bot: add jobs for meson
meson: update available examples to liburing 2.3
Peter Eszlari (1):
add Meson build system
.github/workflows/build.yml | 45 ++++++++++++--
.gitignore | 2 +
examples/meson.build | 19 ++++++
man/meson.build | 116 ++++++++++++++++++++++++++++++++++
meson.build | 119 +++++++++++++++++++++++++++++++++++
meson_options.txt | 14 +++++
src/include/liburing/compat.h.in | 7 +++
src/include/liburing/meson.build | 51 +++++++++++++++
src/include/meson.build | 3 +
src/meson.build | 28 +++++++++
test/meson.build | 219 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
11 files changed, 619 insertions(+), 4 deletions(-)
create mode 100644 examples/meson.build
create mode 100644 man/meson.build
create mode 100644 meson.build
create mode 100644 meson_options.txt
create mode 100644 src/include/liburing/compat.h.in
create mode 100644 src/include/liburing/meson.build
create mode 100644 src/include/meson.build
create mode 100644 src/meson.build
create mode 100644 test/meson.build
Click to show/hide pull request guidelines
Pull Request Guidelines
- To make everyone easily filter pull request from the email
notification, use
[GIT PULL]as a prefix in your PR title.
[GIT PULL] Your Pull Request Title
- Follow the commit message format rules below.
- Follow the Linux kernel coding style (see: https://github.com/torvalds/linux/blob/master/Documentation/process/coding-style.rst).
Commit message format rules:
- The first line is title (don't be more than 72 chars if possible).
- Then an empty line.
- Then a description (may be omitted for truly trivial changes).
- Then an empty line again (if it has a description).
- Then a
Signed-off-bytag with your real name and email. For example:
Signed-off-by: Foo Bar <[email protected]>
The description should be word-wrapped at 72 chars. Some things should not be word-wrapped. They may be some kind of quoted text - long compiler error messages, oops reports, Link, etc. (things that have a certain specific format).
Note that all of this goes in the commit message, not in the pull request text. The pull request text should introduce what this pull request does, and each commit message should explain the rationale for why that particular change was made. The git tree is canonical source of truth, not github.
Each patch should do one thing, and one thing only. If you find yourself writing an explanation for why a patch is fixing multiple issues, that's a good indication that the change should be split into separate patches.
If the commit is a fix for an issue, add a Fixes tag with the issue
URL.
Don't use GitHub anonymous email like this as the commit author:
[email protected]
Use a real email address!
Commit message example:
src/queue: don't flush SQ ring for new wait interface
If we have IORING_FEAT_EXT_ARG, then timeouts are done through the
syscall instead of by posting an internal timeout. This was done
to be both more efficient, but also to enable multi-threaded use
the wait side. If we touch the SQ state by flushing it, that isn't
safe without synchronization.
Fixes: https://github.com/axboe/liburing/issues/402
Signed-off-by: Jens Axboe <[email protected]>
By submitting this pull request, I acknowledge that:
- I have followed the above pull request guidelines.
- I have the rights to submit this work under the same license.
- I agree to a Developer Certificate of Origin (see https://developercertificate.org for more information).
Commit 8ec4e25d7c4212f5eb77cba991ca909410186a1b ("add Meson build system") is missing a Signed-off-by tag from the committer. If you pick someone's else patch, you have to put your Signed-off-by tag as well.
TODOS
- [ ] support the current no Glibc feature provided by the current build system
- [ ] improve meson test tooling for running the tests in a loop (
make runtests-loop)- [ ] replace the custom current build system
- [ ] document the meson build system and its maintainers
- [ ] provide a simple convenient
makewrapper
Should we test the meson build system with GitHub bot too? If yes, please add it to the TODO list please.
Two things from just a quick look:
- We should not retain the old build system. If we're doing meson, we're going all the way
- Exception to the above might be that if I type 'make', I'd like it to just build with meson. This is similar to what eg happens with qemu, and is nice for folks that just expect 'make' to work without needing to know how to build with meson.
provide a simple convenient make wrapper
Ah that was already in the TODO, excellent.
I am thinking about the minimum required meson version.
meson >= 0.60 would allow implementing the current make install-tests
make target using meson installation tags.
Currently we require meson >= 0.57 to ensure support for add_test_setup used
to provide convenient default meson test behavior.
Otherwise the test definitions use the fs meson module requiring at least meson >= 0.54.
Both current requirements can easily be dropped to support older meson versions.
Debian Bullseye for example ships meson 0.56.
Should we consider meson versions included by distributions at all? I consider being compatible with old meson versions not that important since the meson build system is not in the recent liburing-2.2 release and meson, being a standalone python app, is really easy to install without a distro's package manager.
Currently we require meson >= 0.57 to ensure support for
add_test_setupused to provide convenient defaultmeson testbehavior.
It's possible to degrade this on older versions of Meson, by guarding the add_test_setup() in a meson.version() .version_compare(), such that the test setup is only defined on new enough versions of Meson.
On the plus side, this means that you can still configure the project on 0.56.0 using Debian Bullseye's distro-provided Meson. On the negative side, this means that if you try running meson test on such old versions, you don't get that convenient default behavior, but instead run the loop and parallel tests too, which is, uh, not wanted. :D
A hybrid approach could work, if backwards-compatibility is needed. Only with new Meson, define the loop/parallel tests and register the default test setup. Anyone who not only wants to run the tests, but also wants to run them in a loop... will simply be required to run a slightly more modern version of Meson (it's compatible with every single version of python that isn't end of life).
We should not retain the old build system. If we're doing meson, we're going all the way
After pondering this some more, I do believe we should keep the original as well. The point of adding meson should be easier integration for folks. Building liburing is already mostly as trivial as it can be.
That does have the downside of then having two build systems, but I don't think this will be too bad as it's actually pretty simple. It does come with the cost of making sure we stay in sync, we'll just have to see how that goes over time.
But if we retain both, perhaps that can simplify the meson side? We don't really need runtests support, for example, building and installing the core + man pages would really be all that's required. No?
I do believe we should keep the original as well.
From my point of view, it's a good strategy for a while to look at how Meson will work and how folks will adopt it before completely dropping Makefile.
But if we retain both, perhaps that can simplify the meson side? We don't really need runtests support, for example, building and installing the core + man pages would really be all that's required. No?
not really, it's not so complicated and I believe both build systems should be full functionally.
improve meson test tooling for running the tests in a loop
do we have an issue on the meson tracker? I really want to avoid any shell script run from Meson.
A shell script is needed in order to implement custom functionality above and beyond merely executing the compiled test executables. Particularly, grepping dmesg for possible errors. So I think shell scripts are unavoidable (and honestly not a real problem IMO).
If a test should fail if there's something in the dmesg then perhaps that should be incorporated into the test or runner itself so that it returns a negative return value then? That way you can just run meson test --repeat=1000 and have it work automatically. (This is probably out of scope for this PR, just thinking out loud)
improve meson test tooling for running the tests in a loop
do we have an issue on the meson tracker? I really want to avoid any shell script run from Meson.
The runtests-loop make target executes all tests in an infinite loop exiting only if a test failed. Not sure how we would port such behavior to the meson test runner.
I see two possible solutions:
- Add a meson test argument
should_timeoutsimilar toshould_failspecifying that we expect the loop to not exit due to a test failure. @stalkberg do you think I should open an issue on the meson tracker? - Do not implement the looping tests with meson at all. We could simply call the runtests_loop.sh helper from the top-level Makefile.
I think I prefer the second variant. It is easier to implement and does not change the behavior from the status quo.
If a test should fail if there's something in the dmesg then perhaps that should be incorporated into the test or runner itself so that it returns a negative return value then?
This is already the case, that's what the shell script wrapper does.
That way you can just run
meson test --repeat=1000and have it work automatically.
Hmm, good point, this may already work okay like that. Really high numbers is not quite the same as infinite, but for all intents and purposes...
../src/register.c:335:54: error: unused parameter 'flags' [-Werror,-Wunused-parameter]
struct io_uring_buf_reg *reg, unsigned int flags)
we should reduce the strictness of checking or changing API (as I understand we are not interesting in) probably @axboe should look into
Yes, that is deliberately left unused for now, it's so we can avoid adding Yet Another identical function when we do add flags for this.
I'd say keep the unused variables, but add annotations so we can do __maybe_unused to silence them. Then when they do get used, we can just remove that annotation.
Build test / build (x86_64, clang, clang, clang, clang++)
still failed
Sorry about that. The unnecessary explicit setting of the FLAGS in the workflow overrode the -Wno-unused-parameter and -Wno-sign-compare flags.
From https://github.com/fischerling/liburing/commit/111dc30d980abe0126d513f9839f56aab16c258e.patch:
From 111dc30d980abe0126d513f9839f56aab16c258e Mon Sep 17 00:00:00 2001 From: Florian Fischer <[email protected]> Date: Tue, 28 Jun 2022 10:28:13 +0200 Subject: [PATCH] remove obsolte Makefiles Delete the custom build system superseded by meson. Signed-off-by: Florian Fischer <[email protected]> --- .gitignore | 19 -- Makefile | 2 - Makefile.common | 6 - Makefile.quiet | 11 -- configure | 467 ---------------------------------------------- examples/Makefile | 38 ---- liburing.pc.in | 12 -- src/Makefile | 89 --------- test/Makefile | 247 ------------------------ 9 files changed, 891 deletions(-) delete mode 100644 Makefile.common delete mode 100644 Makefile.quiet delete mode 100755 configure delete mode 100644 examples/Makefile delete mode 100644 liburing.pc.in delete mode 100644 src/Makefile delete mode 100644 test/Makefile
No, please keep the original build script and Makefile. Jens has said so too ( https://github.com/axboe/liburing/pull/622#issuecomment-1167843528 ).
I think we should make meson an optional thing rather than a requirement for the build.
Regardless of any other factor, it's usually prudent to implement build system migration as a multi-stage process.
First implement the new build system, then let people try it out and decide whether it suits all their needs, potentially wait a release cycle to give people a chance to shake out any bugs.
And only then decide to either:
- maintain two build systems if desired and people find advantages in each one (the maintenance cost should be minor, it's not that complex of a build)
- declare that the new build system has achieved victory by meeting all the target requirements (including "no one still prefers the other build system"), and it's now safe to remove the old build system
It's surprisingly common for projects to support two build systems, anyway.
Is it possible to updated/rediff that PR against latest version
+ /usr/bin/patch -p1 -s --fuzz=0 --no-backup-if-mismatch -f
1 out of 1 hunk FAILED -- saving rejects to file configure.rej
1 out of 1 hunk FAILED -- saving rejects to file src/Makefile.rej
1 out of 1 hunk FAILED -- saving rejects to file test/Makefile.rej
Just tested updated PR (thank you for that) and looks like meson fails because missing man page file.
+ /usr/bin/meson --buildtype=plain --prefix=/usr --libdir=/usr/lib64 --libexecdir=/usr/libexec --bindir=/usr/bin --sbindir=/usr/sbin --includedir=/usr/include --datadir=/usr/share --mandir=/usr/share/man --infodir=/usr/share/info --localedir=/usr/share/locale --sysconfdir=/etc --localstatedir=/var --sharedstatedir=/var/lib --wrap-mode=nodownload --auto-features=enabled . x86_64-redhat-linux-gnu
The Meson build system
Version: 0.63.0
Source dir: /home/tkloczko/rpmbuild/BUILD/liburing-liburing-2.2
Build dir: /home/tkloczko/rpmbuild/BUILD/liburing-liburing-2.2/x86_64-redhat-linux-gnu
Build type: native build
Project name: liburing
Project version: 2.2
C compiler for the host machine: /usr/bin/gcc (gcc 12.1.1 "gcc (GCC) 12.1.1 20220628 (Red Hat 12.1.1-3)")
C linker for the host machine: /usr/bin/gcc ld.bfd 2.38-17
C++ compiler for the host machine: /usr/bin/g++ (gcc 12.1.1 "g++ (GCC) 12.1.1 20220628 (Red Hat 12.1.1-3)")
C++ linker for the host machine: /usr/bin/g++ ld.bfd 2.38-17
Host machine cpu family: x86_64
Host machine cpu: x86_64
Run-time dependency threads found: YES
Compiler for C supports arguments -Wstringop-overflow=0: YES
Compiler for C supports arguments -Warray-bounds=0: YES
Checking for type "__kernel_rwf_t" : YES
Checking whether type "struct __kernel_timespec" has members "tv_sec", "tv_nsec" : YES
Checking whether type "struct open_how" has members "flags", "mode", "resolve" : YES
Checking if "statx" compiles: YES
Checking if "glibc_statx" compiles: YES
Checking for type "ucontext_t" : YES
Checking for function "makecontext" : YES
Configuring config-host.h using configuration
Configuring compat.h using configuration
man/meson.build:1:0: ERROR: File io_uring_prep_recv_multishot.3 does not exist.
NAK.
# x86 (32-bit) gcc - - arch: i686 + - arch: x86 + cpu: i686 cc_pkg: gcc-i686-linux-gnu cxx_pkg: g++-i686-linux-gnu cc: i686-linux-gnu-gcc [...] - name: Build nolibc run: | - if [[ "${{matrix.arch}}" == "x86_64" || "${{matrix.arch}}" == "i686" ]]; then \ + if [[ "${{matrix.arch}}" == "x86" ]]; then \
You changed i686 to x86, then you removed x86_64 from the nolibc build. Now the bot says:
Run if [[ "x86_64" == "x86" ]]; then \
Skipping nolibc build, this arch doesn't support building liburing
without libc
x86_64 does support building liburing without libc. Why did you remove it from the GitHub bot?
io_uring_prep_recv_multishot.3
Strange since the file was added with 8e182bbdff0f5d6e1640190f713ed11060470b5f. And it works without problems on my system. Could you check if the file does indead not exist?
NAK.
# x86 (32-bit) gcc - - arch: i686 + - arch: x86 + cpu: i686 cc_pkg: gcc-i686-linux-gnu cxx_pkg: g++-i686-linux-gnu cc: i686-linux-gnu-gcc [...] - name: Build nolibc run: | - if [[ "${{matrix.arch}}" == "x86_64" || "${{matrix.arch}}" == "i686" ]]; then \ + if [[ "${{matrix.arch}}" == "x86" ]]; then \You changed i686 to x86, then you removed x86_64 from the nolibc build. Now the bot says:
I changed i686 to x86 because this is the cpu family name used by meson. This makes the creation of the cross compilation file easier.
Run if [[ "x86_64" == "x86" ]]; then \ Skipping nolibc build, this arch doesn't support building liburing without libcx86_64 does support building liburing without libc. Why did you remove it from the GitHub bot?
Good catch thanks! That unintentionally slipped in during the conversion from the old removing the custom build system version.
That unintentionally slipped in during the conversion from the old removing the custom build system version.
OK, the current basic nolibc build looks better. But another question for meson nolibc:
+ - name: Build nolibc with meson + run: | + if [[ "${{matrix.arch}}" == "x86_64" ]]; then \ + rm -r "$MESON_BUILDDIR"; \ + CC=${{matrix.cc}} CXX=${{matrix.cxx}} meson setup "$MESON_BUILDDIR" -Dnolibc=true -Dtests=true -Dexamples=true; \ + ninja -C "$MESON_BUILDDIR"; \ + else \ + echo "Skipping nolibc build, this arch doesn't support building liburing without libc"; \ + fi;
We don't have x86 32-bit for meson nolibc? Forgot to add that?
Strange since the file was added with 8e182bb. And it works without problems on my system. Could you check if the file does indead not exist?
FYI: I'm using as input tar ball autogenerated from giit tag https://github.com/axboe/liburing/archive/liburing-2.2./liburing.-2.2.tar.gz