Option `--runner` stopped working with ShellSpec on v2.2.0

[devarops]

In version 2.2.0, the following command stopped working:

mutmut run --runner "shellspec --shell bash tests"

The tests do not fail and all mutants survive. In the previous version 2.1.0, that command works as expected. I am using Python 3.9.6.

[boxed]

Hmm.. interesting. Can you see that the tests indeed do not fail? That seems strange!

[devarops]

The tests don't fail:

shellspec --shell bash tests

Running: /bin/bash [bash 5.1.4(1)-release]
.....

Finished in 2.26 seconds (user 3.21 seconds, sys 1.74 seconds)
5 examples, 0 failures

mutmut 2.1.0 (Python 3.9.7)

mutmut run --runner "shellspec --shell bash tests"

Legend for output:
🎉 Killed mutants.   The goal is for everything to end up in this bucket.
⏰ Timeout.          Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious.       Tests took a long time, but not long enough to be fatal.
🙁 Survived.         This means your tests needs to be expanded.
🔇 Skipped.          Skipped.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠇ Running...Done

2. Checking mutants
⠧ 8/8  🎉 8  ⏰ 0  🤔 0  🙁 0  🔇 0

mutmut 2.2.0 (Python 3.9.7)

mutmut run --runner "shellspec --shell bash tests"

Legend for output:
🎉 Killed mutants.   The goal is for everything to end up in this bucket.
⏰ Timeout.          Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious.       Tests took a long time, but not long enough to be fatal.
🙁 Survived.         This means your tests need to be expanded.
🔇 Skipped.          Skipped.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠇ Running...Done

2. Checking mutants
⠏ 8/8  🎉 0  ⏰ 0  🤔 0  🙁 8  🔇 0

[devarops]

This behavior persists in version 2.4.3. In version 2.1.0, there are no surviving mutants as expected; in version 2.4.3, all mutants survive.

mutmut 2.1.0 (Python 3.10.6)

mutmut run --runner "shellspec --shell bash tests"

Legend for output:
🎉 Killed mutants.   The goal is for everything to end up in this bucket.
⏰ Timeout.          Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious.       Tests took a long time, but not long enough to be fatal.
🙁 Survived.         This means your tests needs to be expanded.
🔇 Skipped.          Skipped.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠇ Running...Done

2. Checking mutants
⠧ 8/8  🎉 8  ⏰ 0  🤔 0  🙁 0  🔇 0

mutmut 2.4.3 (Python 3.10.6)

mutmut run --runner "shellspec --shell bash tests"

Legend for output:
🎉 Killed mutants.   The goal is for everything to end up in this bucket.
⏰ Timeout.          Test suite took 10 times as long as the baseline so were killed.
🤔 Suspicious.       Tests took a long time, but not long enough to be fatal.
🙁 Survived.         This means your tests need to be expanded.
🔇 Skipped.          Skipped.

mutmut cache is out of date, clearing it...
1. Running tests without mutations
⠇ Running...Done

2. Checking mutants
⠏ 8/8  🎉 0  ⏰ 0  🤔 0  🙁 8  🔇 0

[eivindjahren]

This might be because only exit code 1 is considered a failing test:

https://github.com/boxed/mutmut/blob/8ee19e95ee7f69d374cae82e0d949d5a9dd4b578/mutmut/init.py#L866

I have encountered this before when using custom runners: you have to be sure the error code returned on test failure is 1.

[devarops]

Thank you, @eivindjahren! I'll check the error code and report back.

[devarops]

Indeed, the exit code is 101 for ShellSpec. As a workaround, I used:

mutmut run --runner "shellspec --shell bash tests || exit 1"

Would it be possible to use?:

return returncode == 0

instead of:

https://github.com/boxed/mutmut/blob/8ee19e95ee7f69d374cae82e0d949d5a9dd4b578/mutmut/init.py#L866