boxed
How to specify runner, create JUnit XML in mutmut 3?
Here are my original mutmut CI commands:
mutmut run --no-progress --runner="pytest --assert=plain --exitfirst --randomly-seed=${CI_JOB_ID:?}" ${@+"$@"}"
mutmut junitxml > mutmut.xml
But --no-progress (which is not crucial), --runner, and junitxml are all gone from mutmut 3, and there doesn't seem to be configuration options to achieve the same thing. Another comment mentioned stronger integration with pytest, but I don't see that mentioned in the docs. How can I achieve the same thing with mutmut 3?
Also, a plain mutmut run hits this assertion, which I don't understand. Why treat the name src specially? And what's the fix? Just rename the directory anything else?
junitxml is indeed gone. If I get a PR to reintroduce it, or introduce a separate mutmut write-junit command or something I would merge it. Personally I've never seen the point of it. To me, MT is a tool to improve your tests, not to build reports no one looks at :P
Re src. There's a common error people make with python projects where they have src-style layouts but then import like from src.foo import bar which should be from foo import bar and they need to fix their python path.
Re
src. There's a common error people make with python projects where they havesrc-style layouts but then import likefrom src.foo import barwhich should befrom foo import barand they need to fix their python path.
Whuh? Why would mutmut enforce that specifically for a directory named src only?
I've tried to configure mutmut, but I keep getting weird issues. With the latest state, pytest succeeds, but mutmut fails to run the tests afterwards. Then running pytest after mutmut reveals that the __pycache__ files from mutmut has broken pytest:
Details
❯ pytest
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.12.9, pytest-8.3.3, pluggy-1.5.0
Using --randomly-seed=1329102626
rootdir: /home/victor/my projects/mypy-exercises
configfile: pyproject.toml
plugins: subtests-0.13.1, randomly-3.15.0
collected 38 items
source/annotated/test_all.py ,,,,.,,,,.,,.,,,,.,,,,,,.,,. [ 15%]
source/builtin/test_all.py ,,. [ 18%]
source/generic/test_all.py ,,. [ 21%]
source/optional/test_all.py ,,,,,,.,,.,,.,,,,,,.,,,,,,.,,,,,,. [ 36%]
source/type_checking/test_all.py ,,. [ 39%]
source/generator/test_all.py ,,. [ 42%]
source/why/test_typed_dicts.py . [ 44%]
source/no_return/test_all.py ,,. [ 47%]
source/object_type/test_all.py ,,.,,,,,,,,,,. [ 52%]
source/why/test_complex_types.py ... [ 60%]
source/json_/test_all.py ,,.,,. [ 65%]
source/union/test_all.py ,,,,. [ 68%]
source/callable/test_all.py ,,. [ 71%]
source/instance/test_all.py ,,,,.,,. [ 76%]
source/none/test_all.py ,,. [ 78%]
source/any/test_all.py ,,,,.,,. [ 84%]
source/protocol/test_all.py ..,,,,,,,,. [ 92%]
source/class_/test_all.py ,,.,,.,,. [100%]
=================================================================== 38 passed, 110 subtests passed in 0.11s ====================================================================
↕️ 1 mypy-exercises on renovate/nixpkgs-digest is 📦 v0.2.0 via 🐍 v3.12.9 via ❄️ impure (nix-shell-env)
❯ mutmut run
⠇ Generating mutants
done in 191ms
⠇ Running stats,,.,,.,,.,,,,,,,,,,.,,.uF
=================================================================================== FAILURES ===================================================================================
______________________________________________________ test_should_return_caller_file_io [source.type_checking.exercise] _______________________________________________________
subtests = SubTests(ihook=<_pytest.config.compat.PathAwareHookProxy object at 0x7fc157332e70>, suspend_capture_ctx=<bound method ...ended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test_should_return_caller_file_io>>)
def test_should_return_caller_file_io(subtests: SubTests) -> None:
for function in (exercise, solution):
with (
subtests.test(msg=function.__module__),
function("rb") as function_file_descriptor,
open(__file__, "rb") as self_file_descriptor,
):
> assert function_file_descriptor.read() == self_file_descriptor.read()
E AssertionError: assert b'\nfrom insp..._file\'\n\n\n' == b'from pytest...ptor.read()\n'
E
E At index 0 diff: b'\n' != b'f'
E Use -v to get more diff
source/type_checking/test_all.py:14: AssertionError
------------------------------------------------------------------------------ Captured log call -------------------------------------------------------------------------------
______________________________________________________________________ test_should_return_caller_file_io _______________________________________________________________________
subtests = SubTests(ihook=<_pytest.config.compat.PathAwareHookProxy object at 0x7fc157332e70>, suspend_capture_ctx=<bound method ...ended=False> _capture_fixture=None>>, request=<SubRequest 'subtests' for <Function test_should_return_caller_file_io>>)
def test_should_return_caller_file_io(subtests: SubTests) -> None:
for function in (exercise, solution):
with (
subtests.test(msg=function.__module__),
function("rb") as function_file_descriptor,
open(__file__, "rb") as self_file_descriptor,
):
> assert function_file_descriptor.read() == self_file_descriptor.read()
E AssertionError: assert b'\nfrom insp..._file\'\n\n\n' == b'from pytest...ptor.read()\n'
E
E At index 0 diff: b'\n' != b'f'
E Use -v to get more diff
source/type_checking/test_all.py:14: AssertionError
=========================================================================== short test summary info ============================================================================
[source.type_checking.exercise] SUBFAIL source/type_checking/test_all.py::test_should_return_caller_file_io - AssertionError: assert b'\nfrom insp..._file\'\n\n\n' == b'from...
FAILED source/type_checking/test_all.py::test_should_return_caller_file_io - AssertionError: assert b'\nfrom insp..._file\'\n\n\n' == b'from pytest...ptor.read()\n'
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!! stopping after 2 failures !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
2 failed, 5 passed, 18 subtests passed in 0.34s
failed to collect stats. runner returned 1
❯ pytest
============================================================================= test session starts ==============================================================================
platform linux -- Python 3.12.9, pytest-8.3.3, pluggy-1.5.0
Using --randomly-seed=497393131
rootdir: /home/victor/my projects/mypy-exercises
configfile: pyproject.toml
plugins: subtests-0.13.1, randomly-3.15.0
collected 36 items / 19 errors
==================================================================================== ERRORS ====================================================================================
_______________________________________________________________ ERROR collecting mutants/source/any/test_all.py ________________________________________________________________
mutants/source/any/test_all.py:52: in <module>
SIMPLE_VALUES = [any_integer(), any_string()]
mutants/source/generators.py:141: in any_integer
result = _mutmut_trampoline(x_any_integer__mutmut_orig, x_any_integer__mutmut_mutants, *args, **kwargs)
mutants/source/generators.py:6: in _mutmut_trampoline
mutant_under_test = os.environ['MUTANT_UNDER_TEST']
<frozen os>:714: in __getitem__
???
E KeyError: 'MUTANT_UNDER_TEST'
________________________________________________________________ ERROR collecting source/annotated/test_all.py _________________________________________________________________
import file mismatch:
imported module 'source.annotated.test_all' has this __file__ attribute:
/home/victor/my projects/mypy-exercises/mutants/source/annotated/test_all.py
which is not the same as the test file we want to collect:
/home/victor/my projects/mypy-exercises/source/annotated/test_all.py
HINT: remove __pycache__ / .pyc files and/or use a unique basename for your test file modules
[omitted]
Whuh? Why would mutmut enforce that specifically for a directory named src only?
The vast majority of python projects either have their main module directly in the root, or in a directory called "src". It's fairly common for users or the src-type layouts to have their imports wrong and the project misconfigured because they've cargo culted the directory structure and don't know how python imports work.
Then running pytest after mutmut reveals that the pycache files from mutmut has broken pytest:
No, that's not what you see. Pytest is going into the mutants directory and trying to run tests there like it's a normal source directory, which it isn't. That directory must be excluded from normal pytest runs.
Whuh? Why would mutmut enforce that specifically for a directory named src only?
The vast majority of python projects either have their main module directly in the root, or in a directory called "src". It's fairly common for users or the src-type layouts to have their imports wrong and the project misconfigured because they've cargo culted the directory structure and don't know how python imports work.
So how do I make mutmut work with a project where the source code is in src? Or do I have to rename that directory?
Then running pytest after mutmut reveals that the pycache files from mutmut has broken pytest:
No, that's not what you see. Pytest is going into the
mutantsdirectory and trying to run tests there like it's a normal source directory, which it isn't. That directory must be excluded from normal pytest runs.
Thanks for the tip! I was not able to ignore the directory, but explicitly setting the test directory in pyproject.toml worked:
[tool.pytest.ini_options]
testpaths = ["source"]