cache: take runner into account? / use args for "run"

[blueyed]

I've used mutmut run --use-coverage --runner 'python -m pytest -x testing/test_collection.py' --paths-to-mutate src/_pytest/main.py first.

The first survived mutant was:

% mutmut show 1
EXIT_OK = 0
   |
   V
EXIT_OK = 1

I've then created a test_mutmut.py file, where I've checked that EXIT_OK == 0, but it appears like everything was handled through the cache (according to the speed, there is no actual feedback/count for that).

I assume that using a test file through the --runner argument is considered to be a hack, but I think it is rather important which runner (options) were used, and therefore it should be taken into account for the cache key.

I've seen that there is --tests-dir - maybe that should become --tests-arg?

I've tried using --tests-dir testing/test_mutmut.py, but it appears to run the whole tests?! (the actual command being used would be useful to see with "1. Running tests without mutations" here)

When using run as a click sub-command, it could take any non-option arguments and append them to the --runner command maybe, making --tests-dir obsolete?!

[boxed]

I used to just append the tests-dir to the command, but that presupposes that this is actually valid for all test runners, which is pretty crazy if you think about it. I don't know what the solution here is... maybe that the default runner is "python -m pytest -x {tests_dir}" and then expand before running?

[blueyed]

Is tests_dir required to be separate? Wouldn't a default of python -m pytest -x also work already? I.e. then you would only have "runner".

[boxed]

Well I need to know where the tests are due to how the cache works: for each mutant I store the hash of the tests (as one single hash for the entire dir) it was run against.

[boxed]

You could probably get by by just adding your new test and run mutmut normally. Unless you have lots and lots of mutants and a very slow test suite it might be fine?

[blueyed]

Well, pytest's whole test suite is quite big, that's why I've only run tests for a specific file.

So then the granuality needs to be file-based instead of dir-based maybe?

[boxed]

Yes, that would probably be better but I don't want to have to reimplement passing arguments to all possible runners so I'm hesitant to do that. I did look into getting the actual tests themselves from pytest in order to have even higher granularity but I couldn't figure that out.

If I could get the list of all the tests and their source from the collect phase in pytest that might persuade me to just drop the idea of supporting any other tester because it would be so superior. Also pytest runs nose and unittest anyway so...