ae unit tests don't run on Buildkite

[CyberShadow]

Currently, in D, when building a program and running tests, we don't really have a way to say "please build/run unit tests in this package but not other packages".

This is a problem for libraries, because usually in a library, we only want to run the library's unit tests when we are developing the library, and not any time when the library is simply used as part of a bigger project.

We can sometimes work around this by building the library separately, and then linking to it. Then, the static or shared library being linked will never contain unit test code. However, this doesn't always work - e.g. it doesn't work for templates. However, there isn't really a universal solution.

To work around this issue, ae has all unit test blocks wrapped in a debug (ae_unittest) check, then runs dub test -debug=ae_unittest from its CI. This guarantees that unit tests are only compiled from ae's CI, and never in any programs that use ae.

However, this change had the unfortunate effect that ae's tests also no longer run when other parties do want to run ae's tests. Specifically, because ae is a non-trivial and long-lived collection of D code, and is used in some DFL projects (such as the forum, or the documentation CI system), it is part of the community project test suite (implemented using BuildKite), which tests a collection of community projects for every proposed change in D itself.

I believe this is the script used to test an arbitrary community project: https://github.com/dlang/ci/blob/master/buildkite/travis_get_script

Because it runs dub test without -debug=ae_unittest, it is only able to test a much smaller surface area.

As a consequence of this, the degradation in testing has allowed regressions to be introduced in D that broke ae (and thus projects which use ae): https://github.com/dlang/dmd/issues/21826 . These regressions were only noticed long after the corresponding D release, when I tried to update the compiler to a version which includes these changes.

To resolve this, we need to either:

1. Figure out a better way to make unittest blocks only run when ae itself is being tested, and never when a project using ae is being tested. @Geod24 Could you offer any hints here, is there perhaps a Dub feature I've missed?
2. Improve compatibility with the BuildKite CI scripts so that they build ae with -debug=ae_unittest.

We need to be careful when making these changes to avoid causing disruption for the upstream D CI, even if doing so is simply because we are now exposing missed regressions.

[Geod24]

Dub will not run unittests of dependencies (as discussed here). You can give buildkite any custom build instructions you'd like: https://github.com/dlang/ci/blob/dbf57c5de1e6e5d4fbe898df1b2a9be0577d466f/buildkite/build_project.sh#L231-L235

[CyberShadow]

Dub will not run unittests of dependencies (as discussed here).

But this isn't possible in the general case, right? Especially with templates (i.e. a unit test inside a template, which sometimes makes sense when you want to test every instantiation).

You can give buildkite any custom build instructions you'd like: https://github.com/dlang/ci/blob/dbf57c5de1e6e5d4fbe898df1b2a9be0577d466f/buildkite/build_project.sh#L231-L235

Ah, thanks! And I see there's already a case there that can be tweaked!

[Geod24]

But this isn't possible in the general case, right? Especially with templates (i.e. a unit test inside a template, which sometimes makes sense when you want to test every instantiation).

Correct. IMO this would most likely need to rely on conventions that don't exist yet (such as putting more meaning being the root package than is currently the case), but that's something I personally find desireable.