Test `bash_error_exit_code` May Produce Inaccurate Results

[arapower]

Test bash_error_exit_code uses the binary file that was built just before. If you build the contents of a different branch, the test results will be incorrect.

[Ph0enixKM]

@arapower what does this reference to? I'm having a hard time to understand the issue

[arapower]

@Ph0enixKM Is it okay to have to build before running tests?

[b1ek]

@arapower what does this reference to? I'm having a hard time to understand the issue

if im understanding it right, what ara means is that it currently expects the following workflow: cargo b -> cargo test (which depends on target/debug/amber or whatever)

the problems unfold when the cargo b build is stale or doesn't exist at all

[arapower]

@b1ek Thanks for the explanation. You're right.

[Ph0enixKM]

@arapower @b1ek So what do you propose as a solution to that? I think that the PR creator can check why the test failed and will see that the build crashed or something else

[arapower]

It just seemed strange to me that the test results depended on the built binary. All developers just need to be careful. If that state is acceptable, I think it's fine to leave it as is.

There is a way to work around this. I'm just concerned that it's a workaround.

[b1ek]

There is a way to work around this.

perhaps it could call the main() function or something? im pretty sure that someone has already done something like that

[Ph0enixKM]

I'll mark it with milestone Stable Release for now

[Mte90]

So I fixed a couple of issue for that test on https://github.com/amber-lang/amber/pull/345

Basically the CI was caching the binary between runnings and now run forcing the english language (this in the test itself).

I see also that if the code is changed and you run cargo test it will compile again before the test.

[Mte90]

I think that now the issue is fixed as we compile at every change to Amber the build on testing.