Don't build tests by default in ninja

[dcbaker]

This is a rebased version of #5728, with the changes I requested applied on top. The idea is that tests and benchmarks are removed from the all target, and are placed in their own meson-build-tests and meson-build-benchmarks targets, which the test and benchmark targets depend on. This allows us to avoid building test executables by default.

I'd like to follow this up with a change that allows targets that are not installed to not build by default.

Fixes #2518 Fixes #1704

[lgtm-com[bot]]

This pull request introduces 1 alert when merging aec3478e9d1f634e6ae31c01731bb19e0a28a5ea into 03065f2f008bbc5feb732412c41fc61848a6725f - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a call

[scivision]

Is this a summary of functionality this PR adds?

• I have a "large" project with lots of tests that only devs, not end-users that may build this project are interested in
• whether using "large" directly via meson build or as a Meson subproject, I don't want to build tests by default

this appears to be done new for this PR by:

executable('test_shaky', build_by_default : false)

such that test_shaky is not built by

ninja -C build
# or 
meson install -C build

but would be built by

meson test -C build
# or
ninja -C build test

---

Currently my approach to this is to enclose user-annoying tests like:

if get_option('dev')
  executable('test_shaky')
endif

[dcbaker]

Ideally this would have a follow up of "don't build targets that aren't installed by default", so that if you had helper libraries only used in tests those also wouldn't get built by default. This is for me, motivated by the fact that mesa has a lot of test-only helper libraries that all have to be carefully guarded, and not having to do that guarding would be nice.

[lgtm-com[bot]]

This pull request introduces 1 alert when merging a2985cbb08610a12146f57aa89919bda2ac5e5e9 into 4597235f924fa931a3376814ae76db7e150ded58 - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a call

[dcbaker]

I've adde a bunch of stuff to make working with run_project_tests more pleasant, as I had to run a few tests over and over on windows to figure some of the problems out.

[lgtm-com[bot]]

This pull request introduces 1 alert when merging ef87c6a3ad7b1ec3323241be4a04dafad3aba4d0 into 321774d715f47b3bd3a2e240e0a190dfad7bd22d - view on LGTM.com

new alerts:

• 1 for Wrong number of arguments in a class instantiation

[emersion]

What's the status of this PR? Apart from rebasing, what are the outstanding issues?

[jpakkane]

Featurewise this is ok, but the failing tests need to be fixed at least.

[dcbaker]

I need to find a chance to rebase it and fix the failing tests, just not right at the top of queue right now

[gaudetja]

I need to find a chance to rebase it and fix the failing tests, just not right at the top of queue right now

@dcbaker, @jpakkane, I came across this PR and created https://github.com/mesonbuild/meson/pull/11317 to try and help with rebasing and running tests. Everything seems to be consistent between master and this branch as far as tests passing on my development machine. Thank you for creating this fix years ago!

[evelikov]

@dcbaker can we offer some virtual :cookie: in exchange for updating/landing this PR?

[bonzini]

Closing in favor of #11317, though even that one might be unnecessary given that meson has meson-test-prereq and meson-benchmark-prereq.