Refactor integration tests

[the-horo]

Main takeaways

• make the testsuite work with gdc
• have one test runner that runs all the (possible) tests on as many platforms
• replace the .sh tests with .d programs (in order to run on windows)
• run tests in parallel
• make the test output prettier

---

It took me a while but I think it was worth it.

I've kept the commits to be one test each and the changes are staged to a new directory (new_tests). I don't expect this to be the final form if this PR is merged so I'll keep this as a draft but I hope that this would make reviewing easier.

From my notes while working on this:

• frameworks wasn't testing anything (https://github.com/dlang/dub/pull/3061#issuecomment-3090409283)
• issue2046 wasn't run
• issue2650-deprecated-modules wasn't run
• issue103 is missing a test case (https://github.com/dlang/dub/pull/1177/files#diff-a6b4cf5ac30d4b153861d8ad368ab9cb2aebf25ee28482a6ec201a827ee25a98)
• ddox sometimes decided to ask for confirmation to overwrite files from stdin on windows (making the test stuck). Maybe it had to do with my setup in which I was making changes to how it was run so maybe it's unlikely that a real user would actually hit it. Regardless, I think dub should be preventing the spawned process for having access to stdin

[github-actions[bot]]

✅ PR OK, no changes in deprecations or warnings

Total deprecations: 0

Total warnings: 0

Build statistics:

 statistics (-before, +after)
 executable size=5265672 bin/dub
-rough build time=60s
+rough build time=58s

Full build output

DUB version 1.40.0, built on Jun  7 2025
LDC - the LLVM D compiler (1.41.0):
  based on DMD v2.111.0 and LLVM 20.1.5
  built with LDC - the LLVM D compiler (1.41.0)
  Default target: x86_64-unknown-linux-gnu
  Host CPU: znver3
  http://dlang.org - http://wiki.dlang.org/LDC


  Registered Targets:
    aarch64     - AArch64 (little endian)
    aarch64_32  - AArch64 (little endian ILP32)
    aarch64_be  - AArch64 (big endian)
    amdgcn      - AMD GCN GPUs
    arm         - ARM
    arm64       - ARM64 (little endian)
    arm64_32    - ARM64 (little endian ILP32)
    armeb       - ARM (big endian)
    avr         - Atmel AVR Microcontroller
    bpf         - BPF (host endian)
    bpfeb       - BPF (big endian)
    bpfel       - BPF (little endian)
    hexagon     - Hexagon
    lanai       - Lanai
    loongarch32 - 32-bit LoongArch
    loongarch64 - 64-bit LoongArch
    mips        - MIPS (32-bit big endian)
    mips64      - MIPS (64-bit big endian)
    mips64el    - MIPS (64-bit little endian)
    mipsel      - MIPS (32-bit little endian)
    msp430      - MSP430 [experimental]
    nvptx       - NVIDIA PTX 32-bit
    nvptx64     - NVIDIA PTX 64-bit
    ppc32       - PowerPC 32
    ppc32le     - PowerPC 32 LE
    ppc64       - PowerPC 64
    ppc64le     - PowerPC 64 LE
    r600        - AMD GPUs HD2XXX-HD6XXX
    riscv32     - 32-bit RISC-V
    riscv64     - 64-bit RISC-V
    sparc       - Sparc
    sparcel     - Sparc LE
    sparcv9     - Sparc V9
    spirv       - SPIR-V Logical
    spirv32     - SPIR-V 32-bit
    spirv64     - SPIR-V 64-bit
    systemz     - SystemZ
    thumb       - Thumb
    thumbeb     - Thumb (big endian)
    ve          - VE
    wasm32      - WebAssembly 32-bit
    wasm64      - WebAssembly 64-bit
    x86         - 32-bit X86: Pentium-Pro and above
    x86-64      - 64-bit X86: EM64T and AMD64
    xcore       - XCore
    xtensa      - Xtensa 32
   Upgrading project in /home/runner/work/dub/dub/
    Starting Performing "release" build using /opt/hostedtoolcache/dc/ldc2-1.41.0/x64/ldc2-1.41.0-linux-x86_64/bin/ldc2 for x86_64.
    Building dub 1.39.0-rc.1+commit.266.gcbaa0039: building configuration [application]
     Linking dub
STAT:statistics (-before, +after)
STAT:executable size=5265672 bin/dub
STAT:rough build time=58s

[the-horo]

CI looking good, outside of buildkite which I don't have access to

[thewilsonator]

rm: cannot remove 'test/issue895-local-configuration.sh': No such file or directory

[the-horo]

rm: cannot remove 'test/issue895-local-configuration.sh': No such file or directory

Simply removing that line and changing whatever was used to invoke the test runner with dub run --root test/run_unittest should be good enough. I am trying to get the testsuite to run on as many platforms as it can and I don't want tests that fail on certain setups. It would be best if the tests worked but, if they really can't, they have full control to skip themselves, rather than requiring a user to remove the test directory.

Although, that test is one of the more intrusive ones. It used to overwrite ./etc/dub/settings.json but now it modified a scoped dub.settings.json so it should be safer. It also created a dummy ldc2 named file which can be picked up when invoking dub normally which can pose issues. This is why the test has must_be_run_alone = true, so it can't affect other tests.

---

I did get a few failures when running the testsuite as part of a Gentoo build and one of it seems to be a genuine (minor) bug in the vibe json code but I need more time to make sure it's an actual bug, because I can only reproduce it with a locally built ldc2, and not with dmd, gdc or the upstream releases of ldc2.

[Geod24]

The dlang/ci error is https://github.com/dlang/ci/blob/dc95e2fe07e163e159ae3b5b2bf1efc374fef818/buildkite/build_project.sh#L180-L184

Can get around it by adding -f and removing it once the PR is merged.

[the-horo]

The dlang/ci error is https://github.com/dlang/ci/blob/dc95e2fe07e163e159ae3b5b2bf1efc374fef818/buildkite/build_project.sh#L180-L184

Oh, so that's where it was.

Can get around it by adding -f and removing it once the PR is merged.

buildkite doesn't look like it's running the integration tests, so what's the point in removing those unused test files? Should buildkite be running the full testsuite?

[the-horo]

a genuine (minor) bug in the vibe json code

In regards to the bug: https://github.com/dlang/dub/blob/ad3f246f37044668acc252380758fc08942c4cd4/source/dub/internal/vibecompat/data/json.d#L1452 is calling Json.this(BigInt) https://github.com/dlang/dub/blob/ad3f246f37044668acc252380758fc08942c4cd4/source/dub/internal/vibecompat/data/json.d#L201 which calls zeroFields and then initBigInt https://github.com/dlang/dub/blob/ad3f246f37044668acc252380758fc08942c4cd4/source/dub/internal/vibecompat/data/json.d#L1264 which calls BigInt.opAssign that, in turn, calls BigUint.opAssign. BigUint, however, has an invariant: https://github.com/dlang/phobos/blob/2d8ae67b396f5cc5f4633695c1c5ac67d2bf448e/std/internal/math/biguintcore.d#L240-L244:

    pure invariant()
    {
        assert( data.length >= 1 && (data.length == 1 || data[$-1] != 0 ),
                "Invariant requires data to not empty or zero");
    }

which then fails because the data array has been zero-ed out by zeroFields. As a stack trace:

(gdb) bt
#0  0x00007ffff7f19c20 in _d_assert_msg () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#1  0x00007ffff7c2feae in std.internal.math.biguintcore.BigUint.__invariant() const () from /usr/lib/ldc2/1.40/lib64/libphobos2-ldc-shared.so.110
#2  0x0000555555917f29 in std.internal.math.biguintcore.BigUint.opAssign!(void).opAssign(std.internal.math.biguintcore.BigUint) ()
#3  0x000055555590a60f in std.bigint.BigInt.opAssign!(std.bigint.BigInt).opAssign(std.bigint.BigInt) ()
#4  0x000055555590a191 in dub.internal.vibecompat.data.json.Json.initBigInt() ()
#5  0x000055555590a0f9 in dub.internal.vibecompat.data.json.Json.this(std.bigint.BigInt) ()
#6  0x000055555590e565 in dub.internal.vibecompat.data.json.__unittest_L1450_C7() ()
#7  0x000055555592827e in dub.internal.vibecompat.data.json.__unittest ()
#8  0x00007ffff7f3b7d8 in ?? () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#9  0x00007ffff7f60607 in ?? () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#10 0x00007ffff7f65c98 in rt.sections_elf_shared.DSO.opApply(scope int(ref rt.sections_elf_shared.DSO) delegate) ()
   from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#11 0x00007ffff7f605ac in rt.minfo.moduleinfos_apply(scope int(immutable(object.ModuleInfo*)) delegate) () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#12 0x00007ffff7f4e4ff in object.ModuleInfo.opApply(scope int(object.ModuleInfo*) delegate) () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#13 0x00007ffff7f3b5c8 in runModuleUnitTests () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#14 0x00007ffff7f58bbb in rt.dmain2._d_run_main2(char[][], ulong, extern(C) int(char[][]) function).runAll() () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#15 0x00007ffff7f589d4 in _d_run_main2 () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#16 0x00007ffff7f587ed in _d_run_main () from /usr/lib/ldc2/1.40/lib64/libdruntime-ldc-shared.so.110
#17 0x0000555555bdbea2 in main ()
#18 0x00007ffff774c22e in ?? () from /usr/lib64/libc.so.6
#19 0x00007ffff774c2e9 in __libc_start_main () from /usr/lib64/libc.so.6
#20 0x00005555555d4bc5 in _start ()

You can hit this if the stdlib has been built without -release, which my ldc2 built was, but you can also hit this if you do:

$ DFLAGS='-unittest -link-defaultlib-debug -g' dub test --compiler=ldc2

---

This shouldn't be causing any problems though, since the bigint would be in a valid state after opAssign and the fact that it's invalid as per the invariant doesn't actually break any code. It still needs to be fixed not to trigger the failure though

[the-horo]

Incorporated #3066 and changed issue1531 to use it since I got it to fail on one of my containers where gdc-14 was based on dmd 2.108.1 rather than 2.108.0

I've also updated the tests' README

[the-horo]

As a small benchmark, doing two consecutive runs, the first with a clean repo and no ~/.dub, the second with all the artifacts from the first, with the old and the new test runner, using dmd:

run #	run_unittest -j8	run-unittest.d
1st, clean	5m27.496s	6m45.855s
2nd, cache	2m43.358s	4m21.038s

[the-horo]

Changed run_unittest to compile with -preview=in and -preview=nosharedaccess

[the-horo]

Would like to get this in soon (DConf hackathon perhaps?). Why the move the new_test though ?

I feared that trying to do it in place would make it way harder to do atomic commits for easier review and I also worried that I would miss a dependency on one of the tests. Putting everything in another directory helped with safely porting tests one by one.

This does leave me with having to put everything back into tests. I'm fine doing it in 1 big commit but I think I can manage rebasing each commit to leave everything in /test, if you want it.

[Geod24]

W.r.t. atomic commits, I don't think this is the kind of thing that can be reviewed commit-by-commit anyway. If it was I would prefer if it was split into multiple pull requests, but I don't think it's a good use of yours or my time.

[Geod24]

@the-horo : LMK when this is ready for the next round of review (waiting for files to be moved back).

[the-horo]

@the-horo : LMK when this is ready for the next round of review (waiting for files to be moved back).

Have at it ;)

[the-horo]

Pretty hard to review the whole thing (my browser slows down when I try to scroll down...) but made a few observations.

Yup, the github webui is a pain with so many changes

[Geod24]

Yup, the github webui is a pain with so many changes

Any way we can break it down to smaller (but probably not small) PRs ? I tried to go with marking files as "reviewed" but I can't even scroll when trying to link a file that was removed with the package introducing the code, so I would have to pull this blindly (or review from the CLI, not a great experience either).

[the-horo]

Yup, the github webui is a pain with so many changes

Any way we can break it down to smaller (but probably not small) PRs ? I tried to go with marking files as "reviewed" but I can't even scroll when trying to link a file that was removed with the package introducing the code, so I would have to pull this blindly (or review from the CLI, not a great experience either).

Yeah, I think I can do that.