boxed
"FAILED" after "forced fail test"
I'm trying to run mutmut on the Narwhals project, and have set
[tool.mutmut]
paths_to_mutate = [ "narwhals/_arrow/series.py" ]
tests_dir = [ "tests/expr_and_series/cum_sum_test.py" ]
do_not_mutate= ["tests/utils.py", "narwhals/stable/v1/__init__.py"]
in pyproject.toml
I then run mutmut run and get
⠇ Generating mutants
done in 1068ms
⠧ Running stats
done
⠦ Running clean tests
done
⠴ Running forced fail test...................................................................................................................................................................................s..s [ 12%]
.................................................xxxxxx.................................................................................x............s................................. [ 25%]
...................s..s..........................................................................................s..................................................................... [ 38%]
............x....xxxxxxxs...................x..........................................sss.............x...x.x...x...x.....x......x..x....................s......................xxx..x [ 50%]
xxx.xxx..xx............s...............................................................................................................................................s............... [ 63%]
..........x.....x.xx....x...........x....x...............................x..................s................................................................................x......... [ 76%]
..............xxxxxxxxxxxxxxxxxxxxxxxxx.....................................s...................................................s...........s...............x....xx.................... [ 89%]
............................................................................s.....................................................ss................s....... [100%]
1343 passed, 21 skipped, 73 xfailed in 4.79s
FAILED
I don't understand the output, nor what I'm supposed to do
If I run mutmut browse then it shows 1295 with ? status. I don't know how to interpret that
Any help making sense of how to use mutmut would be appreciated 🙏
The forced fail test should fail immediately. So that any tests are running means it doesn't work. The idea with the forced fail is that all mutants are set to fail so that we can check that mutants can be triggered at all. This is the part that didn't work.
Still.. that some tests fail seems weird too..
The problem is, that mutmut expects paths_to_mutate to contain all relevant files for the project. From my understanding, it is currently implemented as "mutate these projects" rather than "mutate only these files of the project". What you wanted to do with this config, doesn't seem to be possible currently @MarcoGorelli , though I see that it could be useful.
Some background:
Mutmut creates a mutated (more or less) clone of the project at mutants/, but by specifying paths_to_mutate = [ "narwhals/_arrow/series.py" ] it thinks that the project only consists of this single python file. So running pytest in the mutants/ directory would fail because the project is incomplete. Interestingly, we observe the opposite and pytest succeeds even in the forced fail scenario (where every mutation raises an Exception). This is, because pytest does not even find a package in mutants/ and thus all the imports resolve to the original, unmodified code: the tests in mutants/ are executed with the original code rather than the mutated code. This is why the attempted "forced fail" passes all tests.
If possible, I think we should prevent this case where pytest runs the tests in mutants/ with the original code. Maybe we could remove the original directory from sys.path before running the tests. If this works, we would already get the error when trying to run the clean tests, because the imports fail.
Hello from Brazil,
I'm experiencing the same issue.
My test suite has 47 unit tests. When I run the command:
mutmut run
The output is:
(project-service-py3.12) alfredo@machine-address:~/Company/project-service$ mutmut run
⠹ Generating mutants
done in 12ms
⠧ Running stats
done
⠹ Running clean tests
done
⠧ Running forced fail test
****pytest output***
PASSED [100%]
====================================================== 47 passed in 0.29s ======================================================
FAILED
And when I run the command to display the results, it doesn't show any information.
setup.cfg
[mutmut]
paths_to_mutate = ["src"]
runner = "pytest -x -q --import-mode=append tests"
Hi, @alfmorais
This seems to be a different root cause than the original issue, as your paths_to_mutate points to your whole source code.
Can you post a MRE or a link to your repository? If it's similar to this issue, probably your tests in ./mutants/ import your original code, rather than the mutated code in ./mutants/. So some issue with the package setup in ./mutants.
EDIT: Can you try to run mutmut from the current main branch (with python -m pip install --editable <path_to_mutmut_codebase>) and debug=true? If it shows an ImportError, then verify that ./mutants contains all files from your source code.
With #379 mutmut should now have a more reasonable error for cases like this, where the copied source code does not contain the whole package.
And regarding the ambiguity and misuse of paths_to_mutate = [ "narwhals/_arrow/series.py" ], I opened a new issue: #414
@MarcoGorelli On the main branch, narwhal can now be tested with mutmut. I tried it with following config:
[tool.mutmut]
debug = false
tests_dir = [ "tests/expr_and_series/cum_sum_test.py" ]
do_not_mutate= ["tests/utils.py", "narwhals/stable/v1/__init__.py"]
and following invocation:
time mutmut run "narwhals._arrow.series*"
⠦ Generating mutants
done in 21619ms
⠼ Listing all tests
⠙ Running clean tests
done
⠴ Running forced fail test
done
Running mutation testing
⠏ 2103/27668 🎉 36 🫥 2041 ⏰ 10 🤔 0 🙁 16 🔇 0
1.16 mutations/second
real 1m23.737s
user 9m39.184s
sys 0m31.066s
To install mutmut locally, cloning + python -m pip install --editable <path_to_mutmut_codebase> should be enough.
If you have questions, encounter issues or have feedback, feel free to open new issues!