is D4J missing two triggering test cases?

[jose]

Hi René,

I've been running a sanity check on all test methods using $D4J_HOME/framework/lib/formatter.jar and $D4J_HOME/framework/bin/defects4j test -t ..., and there are two test methods to which the outcome of each tool is different:

Lang,64,org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different
Lang,64,org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_equal

for example, the test method testCompareTo_classloader_different passes with $D4J_HOME/framework/bin/defects4j test -t

$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ $D4J_HOME/framework/bin/defects4j compile
$ $D4J_HOME/framework/bin/defects4j test -t org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different

however, it fails if I use $D4J_HOME/framework/lib/formatter.jar:

$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ CP=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
$ $D4J_HOME/framework/bin/defects4j compile
java -cp "$CP:$D4J_HOME/framework/lib/formatter.jar" edu.washington.cs.mut.testrunner.SingleTestRunner org.apache.commons.lang.enums.ValuedEnumTest::testCompareTo_classloader_different

any clue why the custom JUnit runner reports a failing test method and the other approach not?

Extra

What about running the test class from the command line with org.junit.runner.JUnitCore?

$ $D4J_HOME/framework/bin/defects4j checkout -p Lang -v 64b -w Lang-64b
$ cd Lang-64b
$ CP=$($D4J_HOME/framework/bin/defects4j export -p cp.test)
$ $D4J_HOME/framework/bin/defects4j compile
$ java -cp "$CP" org.junit.runner.JUnitCore org.apache.commons.lang.enums.ValuedEnumTest

well, it seems that there are indeed 3 failing test cases and not just 1 as reported by $D4J_HOME/framework/projects/Lang/trigger_tests/64.

and actually, those 3 failing test cases seem to be triggering test cases, as they do not fail on the fixed version, i.e., Lang-64f.

[rjust]

Hi José,

It seems that the difference is caused by an instanceof check on the class loader:

   public void testCompareTo_classloader_equal() throws Exception {                          
        ClassLoader cl = ValuedColorEnum.class.getClassLoader();                              
        if (cl instanceof URLClassLoader) {                                                   
            URLClassLoader urlCL = (URLClassLoader) cl;                                       
            URLClassLoader urlCL1 = new URLClassLoader(urlCL.getURLs(), null);                
            URLClassLoader urlCL2 = new URLClassLoader(urlCL.getURLs(), null);                
            Class otherEnumClass1 = urlCL1.loadClass("org.apache.commons.lang.enums.ValuedColorEnum");
            Class otherEnumClass2 = urlCL2.loadClass("org.apache.commons.lang.enums.ValuedColorEnum");
            Object blue1 = otherEnumClass1.getDeclaredField("BLUE").get(null);                
            Object blue2 = otherEnumClass2.getDeclaredField("BLUE").get(null);                
            assertTrue(((Comparable) blue1).compareTo(blue2) == 0);                           
        }                                                                                     
    }  

The two tests that fail do essentially nothing when executed with Ant.

Executing tests using JUnitCore seems generally reasonable, but this requires more thought as a project's test suite is defined within its build file. What about the following: Let's call JUnitCore from within the run.dev.tests target, for testing purposes, to see whether all defects are still reproducible and whether we observe any more unexpected failing tests. This will give us a better idea of how much effort this change would be.

[jose]

What about the following: Let's call JUnitCore from within the run.dev.tests target, for testing purposes, to see whether all defects are still reproducible and whether we observe any more unexpected failing tests. This will give us a better idea of how much effort this change would be.

Ok, how do we do that?

Cheers, Jose

[rjust]

Currently, the run.dev.tests target calls Ant's junit task with proper classpath settings and the list of tests. We can (for testing purposes) replace the junit task with a call to an exec task that invokes JUnitCore. Converting the classpath and list of tests into the right format should be doable with Ant's filterchains.

What do you think?

[jose]

It sounds reasonable (at least for testing purposes). However, be aware that a solution like that would break $D4J_HOME/framework/bin/defects4j test -t functionality, as org.junit.runner.JUnitCore does not run test methods. (Well, instead of using org.junit.runner.JUnitCore we could use edu.washington.cs.mut.testrunner.SingleTestRunner from $D4J_HOME/framework/lib/formatter.jar, which runs JUnit test classes and/or test methods)

[rjust]

José,

Any chance that you could implement this change so that we can try to reproduce all defects with JUnitCore? Seems like this "only" requires some changes in the top-level build file.

Thanks, René

[Frenkymd]

Hi!

I just came across this issue lately. It is indeed a configuration/execution environment related problem. And, by the way, it can even happen using Ant. One just has to fiddle around a bit with the params of the JUnit Ant task.

I was trying to pass some arguments to the JVM inside the JUnit task therefore I had to make Ant respect my <jvmarg /> tags by setting fork to true (the default case is fork="false").

In the default case, or when you use test with -t then the class loader hierarchy is the following:

• sun.misc.Launcher$ExtClassLoader
• sun.misc.Launcher$AppClassLoader
• AntClassLoader

When you use JUnitCore, or fork is true then the class loader hierarchy is the following:

• sun.misc.Launcher$ExtClassLoader
• sun.misc.Launcher$AppClassLoader

ExtClassLoader and AppClassLoader are URLClassloaders, however AntClassLoader is not, hence both testCompareTo_classloader_different and testCompareTo_classloader_equal will eventually pass in the default case, because the cl instanceof URLClassLoader condition will be false, so the main part of the test will be skipped.