tests/unittests: allow out-of-tree tests

[maribu]

Contribution description

This adds and documents the new EXTERNAL_UNIT_TEST_DIRS environment variable that allows including out-of-tree unit tests. The intention is to allow users of EXTERNAL_MODULE_DIRS to also provide corresponding unit tests and run them all with a single test app.

Testing procedure

Export both EXTERNAL_MDOULE_DIRS and EXTERNAL_UNIT_TEST_DIRS, then build and run the unit tests. The external tests should now be included.

Run again with both variables unset, and only the internal tests should be build and run as before.

Issues/PRs references

None

[riot-ci]

Murdock results

:heavy_check_mark: PASSED

eff6816ded0d0e50e78ca153f34f12d3ff672a8c tests/unittests: allow out-of-tree tests

Success	Failures	Total	Runtime
10936	0	10936	12m:55s

Artifacts

• Documentation preview

[maribu]

What's the benefit of reusing tests/unittests rather than writing your own (out-of-tree) application that contains the unittest?

The motivation is two-fold:

1. The upstream unit tests are pretty good at detecting issues with the toolchain / build setup. So running them along with the downstream tests is useful. And doing so with a single app can speed up time (especially when running the tests on the MCU and the MCU fits both upstream and downstream unit tests: It safes a full build and flash cycle.)
2. The downstream unit test app would be very much a copy-paste of the upstream unit test app. Allowing the upstream app to run the downstream tests instead provides some incentive to upstream fixes/improvements to the unit test app :)

[maribu]

I added a small test app that has both an internal and an external unit test. The python runner checks that two tests are run.

[benpicco]

This needs a rebase.

[crasbe]

I think the build system considers the out-of-tree test folder as a dedicated application:

buechse@skyleaf:~/RIOTstuff/riot-vanillaice/RIOT$ make info-applications | grep tests/build_system/external_unittests
tests/build_system/external_unittests
tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree
tests/build_system/external_unittests/tests-in_tree

And since $(RIOTBASE) is undefined there, the compilation fails:

buechse@skyleaf:~/RIOTstuff/riot-vanillaice/RIOT$ make -C tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree
make: Entering directory '/home/buechse/RIOTstuff/riot-vanillaice/RIOT/tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree'
Makefile:1: /Makefile.base: No such file or directory
make: *** No rule to make target '/Makefile.base'.  Stop.
make: Leaving directory '/home/buechse/RIOTstuff/riot-vanillaice/RIOT/tests/build_system/external_unittests/external_tests_dir/tests-out_of_tree'

You'll probably have to add the folders to makefiles/app_dirs.blacklist.

[maribu]

Good catch. Let's see if this does the trick