junit4

junit4 copied to clipboard

Description

Flaky tests are common occurrences in open-source projects, yielding inconsistent results—sometimes passing and sometimes failing—without code changes. NonDex is a tool for detecting and debugging wrong assumptions on under-determined Java APIs. I have resolved a flaky test issue using NonDex tool, specifically in the MethodSorterTest class located at org.junit.internal.MethodSorterTest.testJvmMethodSorter

Root cause

The flakiness comes from the original codes which tends to sort the list of methods of a class, and use this test to test whether the declared methods of a class is gotten in a predictable order after going through his method sorter. However, the codes here firstly gets the first list of methods using java.lang.Class.getDeclaredMethods(), then gets the second one by making the class of methods going through the sorter, notice that the sorter is written based on a comparator that compare the 'methods' words by getting their hashcode. The fact is, java.lang.Class.getDeclaredMethods() method in Java does not guarantee a specific order for the returned methods. The order in which the methods are returned is not specified and may vary between different Java implementations or versions.

Fix

The logical of comparator is correct, the test is made flaky only because of java.lang.Class.getDeclaredMethods(), thus remove that function to get the methods' list, instead directly apply the comparator's logic in the test to make it work normally.

Setup

Java: openjdk version "11.0.20.1" Maven: Apache Maven 3.6.3

How to test

1. Compile the module mvn install -pl . -am -DskipTests
2. Run regular tests mvn -pl . test -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSorter
3. Run tests with NonDex tool mvn -pl . edu.illinois:nondex-maven-plugin:2.1.1:nondex -Dtest=org.junit.internal.MethodSorterTest#testJvmMethodSorter

After fix, all tests pass