tests: use CTEST_PARALLEL_LEVEL>1 for parallelizing unit tests

[ivotron]

This is currently not possible since using anything more than the default (CTEST_PARALLEL_LEVEL=1) causes some tests to fail. Is this fundamentally difficult to achieve (e.g. because of clashes in net/fs/ipc namespaces)?

[emaxerrno]

Could we farm out duck tape tests to kubernetes?

[emaxerrno]

Or some other scheduler. Maybe time to write our own on top of k8s so we can parallelize them ? Might be an interesting research direction

[BenPope]

This is currently not possible since using anything more than the default (CTEST_PARALLEL_LEVEL=1) causes some tests to fail. Is this fundamentally difficult to achieve (e.g. because of clashes in net/fs/ipc namespaces)?

The fixture tests are parameterised for listener ports, but nearly always use the defaults. Should be a case of making the default a function that allocates any unused port.

I think the filesystem paths are already unique.

[jcsp]

Considering blocking this on https://github.com/vectorizedio/redpanda/issues/2376 (make test runners use faster storage) -- any tests that hit the disk will become significantly less deterministic when they're all competing for the same relatively slow drive in parallel.

[BenPope]

PSA: https://cmake.org/cmake/help/v3.2/prop_test/RUN_SERIAL.html

We could tun on parallel and disable for fixtures. My guess is it won't gain us much, but I mention for posterity.

[BenPope]

Also https://github.com/vectorizedio/redpanda/issues/1970

[gousteris]

@ivotron we didn't consider that as part of optimize redpanda pipeline epic. Should we?

[ivotron]

let's add it as well, if time permits

[jcsp]

Is this fundamentally difficult to achieve (e.g. because of clashes in net/fs/ipc namespaces)?

No, but the tests need some shepherding to make it work.

There are tests (not unit tests really, but things written in C++ that spin up entire redpanda stacks in a single process) that do things like listening on port 9092: those will clash with each other.

Those need to be quarantined from the rest of the true unit tests that can run anywhere anytime. Then the parallelism needs to make sure that those tests don't run in parallel with one another, but that other tests can.

[ivotron]

thanks John. another point that @dotnwat made is that there are some (we counted 7) tests that take ~3mins to execute, because they're bundled in the single binary. if ctest is parallelizing at the level of the binary, then to take advantage of parallel runs we'd need to make sure that those are parallelized well so that they don't run serially, as it'd defeat the purpose of running in parallel

[jcsp]

I did this recently.

[dotnwat]

this same optimization now exists in the meson-based test suite