Mutation analysis hangs for Chart-15

[jose]

Hi @rjust,

@djpaterson reported to me earlier today that mutation analysis does not finish for Chart-15, at least not with a three hours timeout. I have managed to reproduce the issue and here are my findings.

Step-by-step

# Get Java-7
$ cd /tmp
$ wget http://people.cs.umass.edu/~rjust/defects4j/download/jdk-7u80-linux-x64.tar.gz
$ tar -xzf jdk-7u80-linux-x64.tar.gz
$ export JAVA_HOME=/tmp/jdk1.7.0_80
$ export PATH=$JAVA_HOME/bin:$PATH

# Get Defects4J
$ cd /tmp
$ git clone [email protected]:rjust/defects4j.git
$ cd defects4j
$ ./init.sh
$ export D4J_HOME=/tmp/defects4j

# Perform mutation analysis on Chart-15f
$ cd /tmp
$ $D4J_HOME/framework/bin/defects4j checkout -p Chart -v 15f -w Chart-15f
$ cd Chart-15f
$ $D4J_HOME/framework/bin/defects4j mutation

Using all default mutation operators, i.e., AOR + LOR + SOR + COR + ROR + ORU + LVR + STD, the mutation analysis hangs at one particular test suite [junit] MAJOR: 313/324 - org.jfree.chart.plot.junit.PiePlotTests (76ms / 1150): with other sets of mutation operators it either finishes successfully, hangs on the same test suite, or hangs at exporting the kill details. Please find below all the combinations of mutation operators I have tried and what was the outcome.

operators	#mutants	#covered	#killed	hangs at
AOR + LOR + SOR + COR + ROR + ORU + LVR + STD				[junit] MAJOR: 313/324 - org.jfree.chart.plot.junit.PiePlotTests (76ms / 1150):
AOR				[junit] MAJOR: Export mutant kill details to /tmp/Chart-15f/kill.csv
LOR				Cannot mutate project! at /tmp/defects4j/framework/bin/d4j/d4j-mutation line 146
SOR				Cannot mutate project! at /tmp/defects4j/framework/bin/d4j/d4j-mutation line 146
COR	110	102	53
ROR	184	159	66
ORU	1	1	0
LVR	275	240	62
STD	300	239	100
COR + ROR	294	261	119
COR + ROR + ORU	295	262	119
COR + ROR + ORU + LVR	570	502	181
COR + ROR + ORU + LVR + STD	870	741	284
LOR + SOR + COR + ROR + ORU + LVR + STD	870	741	285
AOR + COR				[junit] MAJOR: 297/324 - org.jfree.chart.plot.junit.PiePlotTests (62ms / 522):
AOR + ROR				[junit] MAJOR: Export mutant kill details to /tmp/Chart-15f/kill.csv
AOR + ORU				[junit] MAJOR: Export mutant kill details to /tmp/Chart-15f/kill.csv
AOR + LVR				[junit] MAJOR: 296/324 - org.jfree.chart.plot.junit.PiePlotTests (64ms / 655):
AOR + STD				[junit] MAJOR: Export mutant kill details to /tmp/Chart-15f/kill.csv

It seems that AOR operator or the mutants created by the AOR operator are the reason for mutation analysis to never end. @rjust, any idea of what may be the reason?

-- Best, Jose

[djpaterson]

To add to this - running with all mutation operators on Java 8 produces the same result as just AOR in the table above, the mutation analysis completes but does not get past [junit] MAJOR: Export mutant kill details to /tmp/Chart-15f/kill.csv - the "successful" mutation analysis takes 10-12 minutes to reach this point on my machine (i.e. all tests are completed)

[rjust]

I suspect that a live thread is preventing the JVM from properly shutting down -- after the mutation analysis completed successfully. I'll take a look at this.

In the meantime, you should be able to use the generated data and "kill" the process manually. Not a great solution for automation, though.