One of the Cli test method is flaky (org.apache.commons.cli.BugsTest::test13666)

[Hoyeon9]

One of the Cli test method, org.apache.commons.cli.BugsTest::test13666 is flaky. If test13666 is run alone (using defects4j -t org.apache.commons.cli.BugsTest::test13666) or is executed before other tests that create Options in the configured test suite, it will fail.

The OptionBuilder (src\java\org\apache\commons\cli\OptionBuilder.java) does not initialize the static String argName field, but the OptionBuilder.reset() function changes this value to "arg." The reset() function is called last when an Option object is created with OptionBuilder.create(). test13666 only passes if other tests that create Option objects using OptionBuilder are executed before it. (But the order of test execution is not guaranteed and depends on the suite configuration.)

It must be ensured that the argName in OptionBuilder is initialized to "arg" in test13666 before it create a Option object. To do this, you need to either add .withArgName("arg") as shown in the code below or create a dummy object first.

Option dir = OptionBuilder.withDescription( "dir" )
                                       .hasArg()
                                       .withArgName("arg")
                                       .create( 'd' );

The list below shows the bug versions where this issue occurs. The buggy versions of Cli, 1 2 3 4 5 8 9 10 11 12 17 18 19 20 22 23 24 25 26 27 28 29 30

[rjust]

@Hoyeon9, do you have a concrete use case where this behavior is a problem and a concrete proposal for how to address it?

[Hoyeon9]

Thanks for the reply, @rjust.

I encountered this issue when running a single test method (using the test -t command or external coverage tools). The method fails when executed alone but passes when run as part of the full test class. I temporarily commented out this method to ensure consistent test results, but ideally, the test should be updated (for example, by adding proper initialization at the beginning of the method) or removed, as it behaves like a flaky test.

Actually, I also found similar cases in other projects after posting this issue, suggesting that some tests may depend on shared state or execution order.