poco icon indicating copy to clipboard operation
poco copied to clipboard

Possible flaky tests when building for nixpkgs

Open dwt opened this issue 5 months ago • 2 comments

Describe the bug While packaging for nixpkgs, I found some possible flaky tests.

To Reproduce With nix: nix build github:NixOS/nixpkgs/pull/390150/head#poco should reproduce the problem - if your machine is slow enough. I am not sure how to best trigger this reliable.

Expected behavior We want builds that do not fail in tests that we cannot reproduce locally.

Logs This is what the build server puts out

The following tests FAILED:
          1 - Foundation (Failed)
Errors while running CTest
make: *** [Makefile:91: test] Error 8
error: builder for '/nix/store/7zv191z4wl1agydsiq9c4pv76dgbs3xr-poco-1.14.1.drv' failed with exit code 2;
       last 25 log lines:
       >
       > !!!FAILURES!!!
       > Runs: 842   Failures: 3   Errors: 0
       >
       > There were 3 failures:
       >  1: CppUnit::TestCaller<TimestampTest>.testTimestamp
       >     "d >= 180000 && d <= 300000"
       >     in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/TimestampTest.cpp", line 49
       >  2: CppUnit::TestCaller<ExpireLRUCacheTest>.testExpireN
       >     "aCache.has(1)"
       >     in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/ExpireLRUCacheTest.cpp", line 92
       >  3: CppUnit::TestCaller<UniqueExpireLRUCacheTest>.testExpireN
       >     "aCache.has(1)"
       >     in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/UniqueExpireLRUCacheTest.cpp", line 126
       >
       >
       >
       > 92% tests passed, 1 tests failed out of 12
       >
       > Total Test time (real) = 328.71 sec
       >
       > The following tests FAILED:
       >       1 - Foundation (Failed)
       > Errors while running CTest
       > make: *** [Makefile:91: test] Error 8
       For full logs, run 'nix log /nix/store/7zv191z4wl1agydsiq9c4pv76dgbs3xr-poco-1.14.1.drv'.

Screenshots not applicable

Please add relevant environment information:

  • OS Type and Version ``` ❯ uname -a Darwin Sokrates.local 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:52:00 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6031 arm64
 - POCO Version
 - 1.14.1

**Additional context**
This looks like a flaky test to me, because some process did not answer in time. Do you agree? If you could recommend a workaround or fix that would be great.

I think that in the meantime, I will deactivate these tests, and I would like to link to this bug report to track when we can re-enable them.

dwt avatar Jul 05 '25 19:07 dwt

Forgot to mention the pull request at NixOS/nixpkgs#390150

dwt avatar Jul 05 '25 19:07 dwt

Looking at those tests, they all seem to call sleep(), which is probably just returning way later than expected.

Is there a way to perhaps mock time, so we did not rely on the speed of the machine?

dwt avatar Jul 15 '25 07:07 dwt