Finetune Code Coverage

[TimonPost]

Faultify measures code coverage by injecting a static register function in both the unit test and all methods from the assembly under test. When the code coverage test run is performed those static functions are called. First, the unit test that runs registers its name after that all methods called by this unit test will register their ‘Entity Handle’. After this run, we can exactly see what unit tests covered which methods.

Take the following example:

void MethodUnderTest(bool a) {
    CoverageRegistery.RegisterTarget(32151513351);
}

[Test]
void TestMethod() {
   CoverageRegistery.RegisterTest("SomeNamespace.TestMethod");
   MethodUnderTest(true);
}

To know whether a mutation is covered we need to somehow identify this mutation. We had complications doing so, instead, we register only the method entity handle. This entails that test coverage is a method based not mutation-based which implies that more mutations will be performed.

Take the following example:

void MethodUnderTest(bool a) {
         if (a) {
            // some mutations
         } 
         if (b) {
              // some mutations
         }
}


[Test]
void TestMethod() {
   MethodUnderTest(true);
}

In this case the second if is never executed and also not covered by this test. All mutations in this if can be excluded from the test session.

Implementation

• Injecting a register function before a mutation is difficult to perform in IL-code. Let's take the Addition method IL-code:

.method public hidebysig 
	instance int32 Addition (
		int32 lhs,
		int32 rhs
	) cil managed 
{
	// Method begins at RVA 0x2074
	// Code size 9 (0x9)
	.maxstack 2
	.locals init (
		[0] int32
	)

	// {
	IL_0000: nop
	// return lhs + rhs;

       // ======== THIS IS THE MUTATION PLACE ==========
	IL_0001: ldarg.1      
	IL_0002: ldarg.2      
	IL_0003: add           

	IL_0004: stloc.0
	// (no C# code)
	IL_0005: br.s IL_0007

	IL_0007: ldloc.0
	IL_0008: ret
} // end of method BenchmarkTarget::Addition

Before or after the add opcde we need to inject a function like RegisterTarget(). The problem is that the IL-code before and after varies per instruction. So the complication here is that we might invalidate the IL-logic if we inject our code in the middle of some code.

Todo

Implement mutation-based coverage calculation such that mutations not covered by a test a be excluded which speeds up faultify.

• Figure out how to identify a mutation.
• Figure out how to register this mutation in IL.

Maybe we can have an increasing counter that identifies mutations, in our coverage run we initialize all mutations with those identifiers and after the run, we remove mutations that were not covered.