Possible flaky tests when building for nixpkgs
Describe the bug While packaging for nixpkgs, I found some possible flaky tests.
To Reproduce
With nix: nix build github:NixOS/nixpkgs/pull/390150/head#poco should reproduce the problem - if your machine is slow enough. I am not sure how to best trigger this reliable.
Expected behavior We want builds that do not fail in tests that we cannot reproduce locally.
Logs This is what the build server puts out
The following tests FAILED:
1 - Foundation (Failed)
Errors while running CTest
make: *** [Makefile:91: test] Error 8
error: builder for '/nix/store/7zv191z4wl1agydsiq9c4pv76dgbs3xr-poco-1.14.1.drv' failed with exit code 2;
last 25 log lines:
>
> !!!FAILURES!!!
> Runs: 842 Failures: 3 Errors: 0
>
> There were 3 failures:
> 1: CppUnit::TestCaller<TimestampTest>.testTimestamp
> "d >= 180000 && d <= 300000"
> in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/TimestampTest.cpp", line 49
> 2: CppUnit::TestCaller<ExpireLRUCacheTest>.testExpireN
> "aCache.has(1)"
> in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/ExpireLRUCacheTest.cpp", line 92
> 3: CppUnit::TestCaller<UniqueExpireLRUCacheTest>.testExpireN
> "aCache.has(1)"
> in "/tmp/nix-build-poco-1.14.1.drv-0/source/Foundation/testsuite/src/UniqueExpireLRUCacheTest.cpp", line 126
>
>
>
> 92% tests passed, 1 tests failed out of 12
>
> Total Test time (real) = 328.71 sec
>
> The following tests FAILED:
> 1 - Foundation (Failed)
> Errors while running CTest
> make: *** [Makefile:91: test] Error 8
For full logs, run 'nix log /nix/store/7zv191z4wl1agydsiq9c4pv76dgbs3xr-poco-1.14.1.drv'.
Screenshots not applicable
Please add relevant environment information:
- OS Type and Version ``` ❯ uname -a Darwin Sokrates.local 24.5.0 Darwin Kernel Version 24.5.0: Tue Apr 22 19:52:00 PDT 2025; root:xnu-11417.121.6~2/RELEASE_ARM64_T6031 arm64
- POCO Version
- 1.14.1
**Additional context**
This looks like a flaky test to me, because some process did not answer in time. Do you agree? If you could recommend a workaround or fix that would be great.
I think that in the meantime, I will deactivate these tests, and I would like to link to this bug report to track when we can re-enable them.
Forgot to mention the pull request at NixOS/nixpkgs#390150
Looking at those tests, they all seem to call sleep(), which is probably just returning way later than expected.
Is there a way to perhaps mock time, so we did not rely on the speed of the machine?