[rfc] mutation testing using Stryker Mutator

[brody4hire]

added according to https://stryker-mutator.io/stryker/quickstart (using npx)

with an additional update to only do mutation testing on lib/*.js (with lib/parser/pbxproj.js no longer included)

can be run by the following command: npm run stryker

I posted a browsable (HTML) report based on master ~~(d06078f9e727c2006372e2d06b08d228ece8c9cd)~~ at:

• https://chrisbrody.com/node-xcode-mutation/index.html

I think it would be nice if we could add some more test coverage to kill some of the surviving mutants in pbxFile.js and pbxProject.js.

I think the major drawback is that this can take several hours on a normal workstation or single-CPU server. I was able to get it done in about 25 minutes using a 16-CPU vcore server on vultr.com.

[l3ender]

Would it be possible to merge this change in, especially since we're wanting other issues to resolve/test with Stryker? It wouldn't be part of automated build process at this point (unsure if that's the desired goal, given how long it takes to run), but merging now would at least make manual testing in other branches easier.

[brody4hire]

This proposal is still in draft state, and I would rather wait for some feedback and discussion before merging.

While this would definitely not yet affect automated build processes, it is still something new that could introduce some inconsistency with other packages in our organization where we use nyc (originally "istanbul") to check the code coverage.

This should probably be discussed on the mailing list: https://cordova.apache.org/contact/ or Cordova Slack channel (find it on cordova.io, I am not so active on Slack).

As a side point, I would personally favor moving this project into its own org due to its widespread use outside of Cordova, raised #42 for discussion.

Here is a workaround solution to run Stryker in your workarea while waiting for this proposal:

git remote add brodybits https://github.com/brodybits/cordova-node-xcode.git
git fetch brodybits brodybits-add-stryker
git show 09d7616 | patch -p1
npm install
npm run stryker # or do npx stryker run

A word of warning is that Stryker would be very slow if you don't use something like a 16-core CPU.

[l3ender]

FYI, I ran on my MacBook Pro (2018, 2.6 GHz Intel Core i7, 16 GB 2400 MHz DDR4) and it completed in 22 minutes 52 seconds.

[brody4hire]

I just updated the report: https://chrisbrody.com/node-xcode-mutation/index.html

Here is the old report: https://chrisbrody.com/node-xcode-mutation-2019-09/index.html

Unfortunately I cannot afford the new MacBook Pro upgrade right now.

[l3ender]

I'm having some trouble interpreting the generated HTML reports (new to Stryker). How are you getting the diff view between different test/branch executions (example)?

[brody4hire]

How are you getting the diff view between different test/branch executions (example)?

I will try to walk you through, pardon me if I go into too much detail:

Starting from the index page:

Now click on pbxProject.js and scroll through it.

Let's start with a simple example here:

It shows that if someone removes the contents of the function block for pbxProject.prototype.removeProductFile, the test suite will still pass.

If I click on "274", it will show the function body crossed out:

Next I try removing the lines from the function body and check the results of git diff:

diff --git a/lib/pbxProject.js b/lib/pbxProject.js
index 6466f47..0b660fd 100644
--- a/lib/pbxProject.js
+++ b/lib/pbxProject.js
@@ -143,11 +143,6 @@ pbxProject.prototype.addProductFile = function(targetPath, opt) {
 }
 
 pbxProject.prototype.removeProductFile = function(path, opt) {
-    var file = new pbxFile(path, opt);
-
-    this.removeFromProductsPbxGroup(file);           // PBXGroup
-
-    return file;
 }
 
 /**

and then npm test still passes.

If we look at filetypeForProducttype as discussed in #70:

it shows us several things:

• if we remove the entire function block, it will still psas
• if we replace an object value with a blank string, it will still pass
• if we replace an object key with a blank string, it will still pass

You can try clicking on each of the red numbers to see it more visually.

I hope this is more clear. Please don't hesitate to ask if you have any further questions on this. I think this is a really nice tool in case of code not written according to TDD, better than most other coverage tools.