Address `java.lang.NoClassDefFoundError` in `DefaultClassFilterTest` Through Fixed Order-Dependent Test Execution

[CaseyPan]

trafficstars

Address java.lang.NoClassDefFoundError in DefaultClassFilterTest Through Fixed Order-Dependent Test Execution

Description

When executing the test DefaultClassFilterTest.testDefaultsAreUsedIfOverridesAreGarbage, the intention is to verify the error message when attempting to create a ClassFilter instance with invalid overrides. However, running this test prior to others that rely on the ClassFilter class may lead to a java.lang.NoClassDefFoundError. This error suggests that the JVM previously failed to load the class from the classpath, and subsequently, it won't attempt to load it again, resulting in the error.

To resolve this issue, it is advisable to initialize the ClassFilter successfully before running the specified test. By doing so, the class will be loaded successfully, and subsequent tests can proceed without encountering the NoClassDefFoundError. This sequential approach ensures proper class loading and eliminates the error.

https://github.com/jenkinsci/remoting/blob/1e88c506d547ae7ad99c919396f548a23edfee52/src/test/java/hudson/remoting/DefaultClassFilterTest.java#L101

For these tests, we detected they are Order-Dependent tests by using iDFlakies tool

• hudson.remoting.ClassFilterTest.transport_chunking
• hudson.remoting.ClassFilterTest.transport_nio
• hudson.remoting.ClassFilterTest.transport_non_chunking
• hudson.remoting.ClassFilterTest.userRequest
• hudson.remoting.ClassFilterTest.userRequest_singleClassLoader
• hudson.remoting.DefaultClassFilterTest.testDefaultsNoOverride
• hudson.remoting.DefaultClassFilterTest.testDefaultsOverrideExists
• hudson.remoting.DiagnosedStreamCorruptionExceptionTest.blockingStreamShouldNotPreventDiagnosis
• hudson.remoting.DiagnosedStreamCorruptionExceptionTest.exercise

And we also run the script to find out that the polluter (the test that runs before the test that causes these test to fail) is:

• hudson.remoting.DefaultClassFilterTest.testDefaultsAreUsedIfOverridesAreGarbage

Steps to Reproduce:

1. Clone the repo

   git clone https://github.com/apache/druid.git
   

2. Install the package

   mvn install -am -DskipTests
   

3. Run the test in regular runs

   mvn test -Dtest=src.test.java.hudson.remoting.DefaultClassFilterTest (or any other test)
   

4. The test failed by running in specific order

   mvn test -Dtest=hudson.remoting.DefaultClassFilterTest#testDefaultsAreUsedIfOverridesAreGarbage,ClassFilterTest#transport_chunking (or any other test) -Dsurefire.runOrder=reversealphabetical
   

The following is an example error message from ClassFilterTest.transport_chunking

hudson.remoting.ClassFilterTest.transport_chunking  Time elapsed: 0.014 s  <<< ERROR!
java.lang.NullPointerException
	at hudson.remoting.InProcessRunner.stop(InProcessRunner.java:70)
	at hudson.remoting.ClassFilterTest.tearDown(ClassFilterTest.java:100)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
	at org.junit.internal.runners.statements.RunAfters.invokeMethod(RunAfters.java:46)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:33)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
	at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
	at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
	at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
	at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:364)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:289)
	at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:237)
	at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:428)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:162)
	at org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)

Results:
[INFO] 
[INFO]  Results:
[INFO] 
[ERROR] Errors:
[ERROR] hudson.remoting.ClassFilterTest.transport_chunking
[ERROR] Run 1: ClassFilterTest.transport_chunking:173->setUp:60->setUp:92 » NoClassDefFound C...
[ERROR] Run 2: ClassFilterTest.tearDown:100 » NullPointer
[ERROR] Run 3: ClassFilterTest.transport_chunking:173->setUp:60->setUp:92 » NoClassDefFound C...
[ERROR] Run 4: ClassFilterTest.tearDown:100 » NullPointer
[ERROR] Run 5: ClassFilterTest.transport_chunking:173->setUp:60->setUp:92 » NoClassDefFound C...
[ERROR] Run 6: ClassFilterTest.tearDown:100 » NullPointer
[ERROR] Run 7: ClassFilterTest.transport_chunking:173->setUp:60->setUp:92 » NoClassDefFound C...
[ERROR] Run 8: ClassFilterTest.tearDown:100 » NullPointer
[ERROR] Run 9: ClassFilterTest.transport_chunking:173->setUp:60->setUp:92 » NoClassDefFound C...
[ERROR] Run 10: ClassFilterTest.tearDown:100 » NullPointer
[INFO] 
[INFO] 
[ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO]  Total time:  12.325 s
[INFO] Finished at: 2023-12-08T01:21:16-06:00

5. Tests passed after the fix

[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.223 s -- in hudson.remoting.ClassFilterTest
[INFO] 
[INFO] Results:
[INFO] 
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  4.739 s
[INFO] Finished at: 2023-12-10T15:28:00-06:00
[INFO] ------------------------------------------------------------------------

Testing done

### Submitter checklist
- [x] Make sure you are opening from a **topic/feature/bugfix branch** (right side) and not your main branch!
- [x] Ensure that the pull request title represents the desired changelog entry
- [x] Please describe what you did
- [x] Link to relevant issues in GitHub or Jira
- [x] Link to relevant pull requests, esp. upstream and downstream changes
- [x] Ensure you have provided tests - that demonstrates feature works or fixes the issue

[jglick]

Thanks for finding this! Unclear to me that the modified test is still asserting what aa4901c5f85e62a3617222bda42cf00deec4de2f intended, whatever that was. The test should perhaps be rewritten to use some more internal method rather than expecting an Error, or perhaps the test cases should be broken up into one per class each run in its own JVM, etc.