Make a multi-threaded test harness

[odow]

Discussed at the JuMP-dev '25 hackathon.

• The tests take a long time to run
• There is much parallelism available
• We've already organised the tests into standalone submodules
• Each submodule has a set of independent test functions
• It should be easy to run both the submodules in parallel, and the tests within a submodule in parallel

We currently use an environment variable to partition the CI tests: https://github.com/jump-dev/MathOptInterface.jl/blob/034a5272dad29a233783cc49ed7e15195450dd96/test/runtests.jl#L7-L70

https://github.com/jump-dev/MathOptInterface.jl/blob/034a5272dad29a233783cc49ed7e15195450dd96/.github/workflows/ci.yml#L25-L30

But there's a cost: running the tests in separate jobs like this incurs the same compilation overhead. Perhaps multithreading within the same Julia process would be more efficient.

I started working on something like this for JuMP:

https://github.com/jump-dev/JuMP.jl/blob/master/test/Kokako.jl

It was based on https://github.com/IainNZ/Pukeko.jl. If we made progress, we could take over Iain's repo. This would be useful across the broader Julia ecosystem.

[odow]

Okkkaay. I looked into this again, and I remember where I got stuck.

Variants of this:

function runtests()
    filter_fn(name) = startswith("$name", "test_")
    Threads.@threads for name in filter!(filter_fn, names(@__MODULE__; all = true))
        getfield(@__MODULE__, name)
    end
    return
end

don't work because allocations change (0 when run normally, many with threading). I must need to sprinkle some magic dust around.

[blegat]

We could maybe consider using https://github.com/julia-vscode/TestItems.jl

[odow]

Or https://github.com/JuliaTesting/ParallelTestRunner.jl