muter
muter copied to clipboard
muter-mutation-testing
Muter takes 1000+ minutes to run and once computer goes to sleep it doesn't progress.
I am running Muter on a fairly large project with around 300 swift files. I also tried to ran muter for a specific directory using --files-to-mutate which has around 15 swift files. Both the times, it says ETC is 1000+ minutes. Is anybody else seeing this issue or suggestions to make muter run faster?
I wonder if we could grab a power assertion to keep the computer awake? As for the speed, the best thing you can do in your own code is to do whatever you can to make your test suite run faster. Eliminating waits and async stuff will help, as will breaking your app up into frameworks so the cycle to build and run particular test files is shorter after a file is changed.
@itsmeabhinay and myself felt like a runtime of 1000+ mins was not/couldnt be right and figured we had some config issue or maybe project settings that were correct which caused such a long runtime. As a next step, we created a brand new sample app, added 2 classes and 10 Unit tests. The runtime for this sample app, was still 1hr.
My main question here is - does it seem reasonable/does 60mins to run mutation tests on 10 unit tests seem correct? is that what others are seeing? An equivalent mutation test plugin on java/kotlin android takes 2mins to run - i know the platforms are different and run times will be too, but 1hr to mutation test 10 tests seems crazy?
Agreed. That seems high! I don't have any concrete benchmarks for you offhand, but I'd dig into how long xcodebuild or swift test takes to run the same tests as a starting point.
@itsmeabhinay wanna grab some stats on build and run times and post them here? Also thinking is it an xcode version related thing? Did we try make the sample app in xcode12 and see if the run time is the same? @ZevEisenberg What's a birthday runtime on projects you've applied muter to?
I don't remember offhand, but I seem to recall it being pretty close to a clean or perhaps incremental build (I forget which, and it's an important difference) multiplied by the number of mutants found. The disk operations to swap out the mutated files are pretty negligible unless it's a massive project, so it's nearly all build time.
@itsmeabhinay I'm closing this one for now since the mutation schemata strategy has been deployed. Please give it a try, it should improve a lot Muter's speed of execution.
