[BUG] Continuous integration sometimes generates false "test failed" notifications

[hdwalters]

I recently received a random "Cargo & Clippy Test" email on my glob function branch, directing me to https://github.com/amber-lang/amber/actions/runs/10872830349/job/30168293449. This complains about a failed unit test, which I haven't touched:

---- tests::stdlib::test_stdlib_src_tests_stdlib_abs_ab stdout ----
thread 'tests::stdlib::test_stdlib_src_tests_stdlib_abs_ab' panicked at src/tests/mod.rs:18:28:
assertion `left == right` failed
  left: ""
 right: "1\n1"

I do not see how this unit test could possibly fail, as it doesn't access the file system or anything else external:

$ cat src/tests/stdlib/abs.ab 
import * from "std/math"

// Output 
// 1
// 1

main {
    echo abs(1);
    echo abs(-1);
}

[b1ek]

probably related to bash version

[hdwalters]

probably related to bash version

If that were the problem, surely it would succeed or fail consistently? I would expect the CI server to do something like spinning up a Docker instance, to run tests in a consistent environment.

[Mte90]

So it is a bug that happens sometimes. Re run the jobs fixes it, I think that https://github.com/amber-lang/amber/pull/381 could fix it but that PR is blocked because there is an issue in escaping with bash generated.

[Mte90]

I think that now postprocessor was fixed this issue is gone. Just keep the ticket open for a while to close it later.

[hdwalters]

I think that now postprocessor was fixed this issue is gone. Just keep the ticket open for a while to close it later.

Please don't close it just yet; it seems to be an intermittent problem. I've occasionally had test failures on my own PC when I run cargo test, but these go away if I run cargo build first, especially when I'm switching Git branches.

[hdwalters]

Also, surely the absence of a postprocessor binary on the CI server would not cause intermittent failures as described above; it would either always succeed, or always fail.

[Mte90]

can we close this? I don't see anymore the issue

[hdwalters]

It still occurs intermittently for me, though in different circumstances; but I have no idea why. I guess you can close the issue if it's not going to get investigated.

[Mte90]

I think that is the case to close it, I am not able to replicate it in my machine and in the official CI.