boxed
Performance optimizations
Currently I add the following to setup.cfg:
[mutmut]
runner=sh -c 'pytest --exitfirst --inc "$@" || [ $? = 5 ]' -
However it would be good to have enabled these by default. Other optimizations are also possible e.g.:
- not running new Python process for every run - some libraries e.g. Tensorflow load quite slowly
- parallel execution
The current default runner is:
python -m pytest -x --assert=plain
where the -x option is shorthand for --exitfirst, so mutmut already covers this.
The --inc option is something which is provided by the pytest-incremental plugin.
This is definitely helpful to increase the speed of mutation testing, but as this requires an additional plugin I think we should not add it to the default option.
However, I just recently expanded the mutmut readme with the same thought about optimizing execution time in mind.
I did not know about pytest-incremental, but it looks like it would be worth noticing there, as it seems to be the better alternative to what I described in selection based on imports!
Parallel execution is quite tricky and not starting a new interpreter is basically not possible. I have spent maaaaany hours trying to get that to work. It's just not reliable.
The plan is to switch to some kind of in memory mutation schema with fork. That would do all that but it limits mutmut to Linux only (no windows or macos). It would also be a ton of work which I havet had time to actually do :)
Limiting to Linux only is not a big problem since we have things like Docker or Windows Services For Linux 2 :-)
As for --inc option and pytest-incremental dependency: you can somehow detect when to apply it (e.g. when pytest-incremental is installed, or when mutmut --inc is invoked. It's still better than runner=sh -c 'pytest --exitfirst --inc "$@" || [ $? = 5 ]' - - for people less skilled with BASH this command can be too hard to figure out :-)
Hm, I would be glad if at least there would be a fallback mechanism for sequential execution to still support Windows and macOS.
@iirekm why is the "$@" || [ $? = 5 ]' - necessary?
If I understand correctly you use this to say "run the tests but if you don't find any that's also OK" (as exit code 5 is "no tests collected")?
@DudeNr33 yea.. I've been a mac user for many many years, so I also don't like the idea of dropping macos support.
Yes. Unfortunately without this checking for exit code 5, every mutation was reported as killed, which wasn't true.
Installing docker desktop for Mac takes literally 2 minutes. For users who don't need parallel run, the current method can be used. Maybe it can be somehow implemented natively with Mac apis (they implement a subset of linux/posix apis)
@iirekm Please don't dismiss the problems introduced by docker. And having two vastly different execution models is a big source of bugs, development time sinks and troubleshooting problems. It's better not to do that.
Having 3 separate impls (one for lx, one for win, one for Mac) is even bigger risk of bugs 😔 Maybe the thing you want to do is done by somebody in form of a library, ported to all 3 systems?
pt., 8 paź 2021, 13:32 użytkownik Anders Hovmöller @.***> napisał:
@iirekm https://github.com/iirekm Please don't dismiss the problems introduced by docker. And having two vastly different execution models is a big source of bugs, development time sinks and troubleshooting problems. It's better not to do that.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/boxed/mutmut/issues/236#issuecomment-938568157, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGAYE6FZRCJG65JEYDAOZLUF3JDBANCNFSM5FRKLGFQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.
Having 3 separate impls (one for lx, one for win, one for Mac) is even bigger risk of bugs
Yea.. and that's why we only have one implementation. I'm confused.. what are you talking about now?
I am probably just not fit enough in Bash scripting, but the || [ $? = 5 ] would lead to a "total" exit code of 0 if no tests where ran, correct?
In that case a mutant would be reported as killed if no tests where run, or am I missing something?