Add Mutation Testing

[ahawkins]

Current work in progress with @mbj

[ahawkins]

@mbj So I'm almost done testing ::Harness only. Can't seem to get to the 100% mark though.

Here's the current report:

adam at mba in ~/p/harness(mutant*) bundle exec rake mutant
bundle exec mutant -I lib -r harness --use minitest '::Harness'
Mutant configuration:
Matcher:         #<Mutant::Matcher::Scope cache=#<Mutant::Cache> scope=Harness>
Strategy:        #<Mutant::Minitest::Strategy>
Expect Coverage: 100.000000%
Harness.collector:/Users/adam/projects/harness/lib/harness.rb:49
......
(06/06) 100% - 0.65s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:10
...FFFF.
(04/08)  50% - 1.25s
Harness.count:/Users/adam/projects/harness/lib/harness.rb:22
...................................
(35/35) 100% - 5.06s
Harness.decrement:/Users/adam/projects/harness/lib/harness.rb:18
...............................
(31/31) 100% - 4.53s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37
........................F....................F.........F.
(54/57)  94% - 5.93s
Harness.gauge:/Users/adam/projects/harness/lib/harness.rb:41
...................................
(35/35) 100% - 5.10s
Harness.increment:/Users/adam/projects/harness/lib/harness.rb:14
...............................
(31/31) 100% - 4.50s
Harness.queue:/Users/adam/projects/harness/lib/harness.rb:45
......
(06/06) 100% - 0.63s
Harness.time:/Users/adam/projects/harness/lib/harness.rb:30
............................................
(44/44) 100% - 6.33s
Harness.timing:/Users/adam/projects/harness/lib/harness.rb:26
...................................
(35/35) 100% - 4.98s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:10
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:2b302
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @s53eff8ffac026cdd4412 ||= Config.new
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:e076e
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= Config
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:b6fc6
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= nil.new
 end
evil:Harness.config:/Users/adam/projects/harness/lib/harness.rb:10:b452c
@@ -1,4 +1,4 @@
 def self.config
-  @config ||= Config.new
+  @config ||= nil
 end
(04/08)  50% - 1.25s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37
evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:37:48223

(54/57)  94% - 5.93s
Subjects:  10
Mutations: 288
Kills:     281
Alive:     7
Runtime:   39.07s
Killtime:  38.95s
Overhead:  0.29%
Coverage:  97.57%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -I lib -r harness --use...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

As you can see it's doing mutations to the config method. I added a test to assert that Harness.config always returns an instance of Harness::Config. Am I doing something wrong? I tested that it does in fact run all the tests and that is the case.

Secondly, is there any possible way to fix or ignore this mutation?

 def self.config
-  @config ||= Config.new
+  @s53eff8ffac026cdd4412 ||= Config.new
 end

[mbj]

@ahawkins I have a ticked for these. This mutation will be removed. Or at least be more intelligent on them. I personally prefer to refactor the need for ||= away, that also reduces some nasty races.

[ahawkins]

@mbj how do you refactor them?

[solnic]

blah @mbj you still didn't remove those? ;)

@ahawkins you can either refactor the code to not use ||= (which IMHO is a good thing to do) or just ignore those mutations

I explained (more or less) why I don't like ||= in such cases here but this is probably a broader topic related to things like using objects not classes, injecting dependencies, avoiding mutable state and maybe something else ;)

[mbj]

@ahawkins There is no general advice for all ||=.

Specific advice is possible:

def self.config
  @config ||= Config.new
end

You lazy initialize a mutable object under a global accessible location.

If you do this:

class Config
  INSTANCE = new
end

def self.config
  Config::INSTANCE
end

The mutant is dead.

Has the same public interface semantics. With less code, and race free.

I generally would not use global mutable state at all. But that is another topic.

[ahawkins]

@mbj thanks.

Next question. Almost to 100% across the project. Cannot get AsyncQueue correct though. Is there some problem with attr_reader?

(11/11) 100% - 1.59s
Cannot find definition of: Harness::AsyncQueue#queue in /Users/adam/projects/harness/lib/harness/async_queue.rb:5
Harness::AsyncQueue#collector:/Users/adam/projects/harness/lib/harness/async_queue.rb:25
evil:Harness::AsyncQueue#collector:/Users/adam/projects/harness/lib/harness/async_queue.rb:25:9b3ef
@@ -1,4 +1,4 @@
 def collector
-  Harness.collector
+  Harness
 end
(05/06)  83% - 0.76s
Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:f05b8
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @s0c72b44866a20ed13aff = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:5f13a
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @queue = Queue
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:88a5f
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @queue = nil
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:52267
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  nil
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:f18f1
@@ -1,12 +1,11 @@
 def initialize
-  @queue = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
evil:Harness::AsyncQueue#initialize:/Users/adam/projects/harness/lib/harness/async_queue.rb:7:daeda
@@ -1,12 +1,12 @@
 def initialize
-  @queue = Queue.new
+  @s613e120fb7c7fde109d5 = Queue.new
   Thread.new do
     loop do
       msg = queue.pop
       method_name = msg.first
       args = msg.last
       collector.__send__(method_name, *args)
     end
   end
 end
(55/61)  90% - 8.05s
Subjects:  3
Mutations: 78
Kills:     71
Alive:     7
Runtime:   10.46s
Killtime:  10.40s
Overhead:  0.55%
Coverage:  91.03%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -I lib -r harness --use...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

I see mutant is saying cannot find definition for queue method. Should I define methods myself instead of using attr_reader?

[mbj]

@ahawkins The warnings about "Cannot find definition of Foo#bar" will are not really warnings. I added these at the time the matcher was not as good as today. I'll only emit these warnings in future when you explicitly asked for Foo#bar. Mutant does not know where to find ASTs for methods generated via attr_reader or definie_method in its current execution model.

You can savely ignore these warnings, they'll go away soon.

[ahawkins]

@mbj c5cc8a3 was definitely the hardest one yet. This is an interesting experience to say the least :)

[mbj]

@ahawkins Writing code in a way thats easily mutation testable is hard. But it pays back. Does not matter what you see on twitter. People think I prefer local maxima.

[mbj]

@ahawkins I'm busy. Cannot expand the following: Options for testing Threaded code:

• Inject a predicable Thread mock. (Allows most scenarios to be steered explicitly!)
• Ignore the Thread subject in mutation testing. But move as much code that gets executed in the thread to a mutation testable unit.
• (IDEA) never tried. Create a thread mock thats sigle steppable via debug hooks.

[solnic]

@mbj what do you mean by local maxima in this context?

[mbj]

@solnic Anima::Builder was a local maxima. Where I did not spotted a better local maxima the new code. Dunno if I can ever prove we reached a global maxima of beautifulness. See linear optimization speak.

[ahawkins]

@mbj Only 3 mutations remaining the entire code base. I'm not sure what to do about them. They are "evil" mutants. The mutant output does not include any code changes though:

adam at mba in ~/p/harness(mutant*) bundle exec rake mutant
bundle exec mutant -d -I lib -r harness --use minitest '::Harness'
Mutant configuration:
Matcher:         #<Mutant::Matcher::Scope cache=#<Mutant::Cache> scope=Harness>
Strategy:        #<Mutant::Minitest::Strategy>
Expect Coverage: 100.000000%
Harness.collector:/Users/adam/projects/harness/lib/harness.rb:51
......
(06/06) 100% - 1.01s
Harness.config:/Users/adam/projects/harness/lib/harness.rb:12
....
(04/04) 100% - 0.77s
Harness.count:/Users/adam/projects/harness/lib/harness.rb:24
...................................
(35/35) 100% - 10.11s
Harness.decrement:/Users/adam/projects/harness/lib/harness.rb:20
...............................
(31/31) 100% - 9.26s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39
........................F....................F.........F.
(54/57)  94% - 7.81s
Harness.gauge:/Users/adam/projects/harness/lib/harness.rb:43
...................................
(35/35) 100% - 10.04s
Harness.increment:/Users/adam/projects/harness/lib/harness.rb:16
...............................
(31/31) 100% - 9.28s
Harness.queue:/Users/adam/projects/harness/lib/harness.rb:47
......
(06/06) 100% - 0.97s
Harness.time:/Users/adam/projects/harness/lib/harness.rb:32
............................................
(44/44) 100% - 12.79s
Harness.timing:/Users/adam/projects/harness/lib/harness.rb:28
...................................
(35/35) 100% - 9.90s
Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39
evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

evil:Harness.delta:/Users/adam/projects/harness/lib/harness.rb:39:48d86

(54/57)  94% - 7.81s
Subjects:  10
Mutations: 284
Kills:     281
Alive:     3
Runtime:   72.04s
Killtime:  71.93s
Overhead:  0.15%
Coverage:  98.94%
Expected:  100.00%
rake aborted!
Command failed with status (1): [bundle exec mutant -d -I lib -r harness --...]
/Users/adam/projects/harness/Rakefile:11:in `block in <top (required)>'
Tasks: TOP => mutant
(See full trace by running task with --trace)

[mbj]

@ahawkins Catching up on a 4 y old thread. mutant-minitest was just merged to master. Do you want to retry? I'd be happy to advice.