com.hazelcast.jet.elastic.impl.ElasticSourcePTest failures

[olukas]

master (commit 4e57bfb95e70a8f56e865edbe7b8c8e2a48bdb04)

Failed on IBM JDK 8: http://jenkins.hazelcast.com/job/jet-oss-master-ibm-jdk8/250/testReport/com.hazelcast.jet.elastic.impl/

It seems that mention below exception in com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithOptionsFn_then_shouldUseOptionsFnForScrollRequest causes failure in other test in test class:

• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.givenMultipleResults_when_runProcessor_then_useScrollIdInFollowupScrollRequest
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.given_singleHit_when_runProcessor_then_produceSingleHit
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithCoLocationAndSlicing_thenUseSlicingBasedOnLocalValues
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithCoLocation_thenSearchShardsWithPreference
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithCoLocation_then_useLocalNodeOnly
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithOptionsFn_then_shouldUseOptionsFnForSearchRequest
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessorWithParallelism_thenUseSlicingBasedOnGlobalValues
• com.hazelcast.jet.elastic.impl.ElasticSourcePTest.when_runProcessor_then_executeSearchRequestWithScroll

Stacktrace:

java.lang.IllegalStateException: Could not initialize plugin: interface org.mockito.plugins.MockMaker (alternate: null)
	at org.mockito.internal.configuration.plugins.PluginLoader$1.invoke(PluginLoader.java:74)
	at com.sun.proxy.$Proxy5.isTypeMockable(Unknown Source)
	at org.mockito.internal.util.MockUtil.typeMockabilityOf(MockUtil.java:29)
	at org.mockito.internal.util.MockCreationValidator.validateType(MockCreationValidator.java:22)
	at org.mockito.internal.creation.MockSettingsImpl.validatedSettings(MockSettingsImpl.java:240)
	at org.mockito.internal.creation.MockSettingsImpl.build(MockSettingsImpl.java:228)
	at org.mockito.internal.MockitoCore.mock(MockitoCore.java:61)
	at org.mockito.Mockito.mock(Mockito.java:1908)
	at org.mockito.Mockito.mock(Mockito.java:1880)
	at com.hazelcast.jet.elastic.impl.ElasticSourcePTest.setUp(ElasticSourcePTest.java:73)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runners.Suite.runChild(Suite.java:128)
	at org.junit.runners.Suite.runChild(Suite.java:27)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.apache.maven.surefire.junitcore.JUnitCore.run(JUnitCore.java:55)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:137)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeLazy(JUnitCoreWrapper.java:119)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:87)
	at org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:75)
	at org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:158)
	at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
	at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
	at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
	at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
Caused by: java.lang.IllegalStateException: Failed to load interface org.mockito.plugins.MockMaker implementation declared in java.lang.ClassLoader$CompoundEnumeration@7194765f
	at org.mockito.internal.configuration.plugins.PluginInitializer.loadImpl(PluginInitializer.java:54)
	at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:57)
	at org.mockito.internal.configuration.plugins.PluginLoader.loadPlugin(PluginLoader.java:44)
	at org.mockito.internal.configuration.plugins.PluginRegistry.<init>(PluginRegistry.java:22)
	at org.mockito.internal.configuration.plugins.Plugins.<clinit>(Plugins.java:19)
	at org.mockito.internal.util.MockUtil.<clinit>(MockUtil.java:24)
	... 42 more
Caused by: org.mockito.exceptions.base.MockitoInitializationException: 
Could not initialize inline Byte Buddy mock maker. (This mock maker is not supported on Android.)

Java               : 1.8
JVM vendor name    : IBM Corporation
JVM vendor version : 2.9
JVM name           : IBM J9 VM
JVM version        : 8.0.5.40 - pxa6480sr5fp40-20190807_01(SR5 FP40)
JVM info           : JRE 1.8.0 Linux amd64-64-Bit Compressed References 20190802_424001 (JIT enabled, AOT enabled)
OpenJ9   - 106f6ce
OMR      - fe07f6f
IBM      - af2a365
OS name            : Linux
OS version         : 3.10.0-693.11.1.el7.x86_64

	at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.<init>(InlineByteBuddyMockMaker.java:168)
	at java.lang.J9VMInternals.newInstanceImpl(Native Method)
	at java.lang.Class.newInstance(Class.java:1848)
	at org.mockito.internal.configuration.plugins.PluginInitializer.loadImpl(PluginInitializer.java:49)
	... 47 more
Caused by: java.lang.IllegalStateException: Error during attachment using: net.bytebuddy.agent.ByteBuddyAgent$AttachmentProvider$Compound@ed6b9225
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:427)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:401)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:353)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:330)
	at org.mockito.internal.creation.bytebuddy.InlineByteBuddyMockMaker.<clinit>(InlineByteBuddyMockMaker.java:104)
	... 50 more
Caused by: java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:90)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
	at java.lang.reflect.Method.invoke(Method.java:508)
	at net.bytebuddy.agent.Attacher.install(Attacher.java:99)
	at net.bytebuddy.agent.ByteBuddyAgent.install(ByteBuddyAgent.java:422)
	... 54 more
Caused by: com.sun.tools.attach.AttachNotSupportedException: target not found
	at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTarget(OpenJ9VirtualMachine.java:120)
	at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:59)
	at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:43)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	... 60 more

[frant-hartm]

This looks like a compatibility issue of Mockito (and underlying ByteBuddy library) and OpenJ9.

This is a similar, but not identical, issue https://github.com/mockito/mockito/issues/1899.

It suggests it could be caused by another agent (e.g PowerMockito which we also have on the classpath) or some classpath scanning order.

I tried to run this locally on OpenJ9 around 100 times, but the failure didn't occur. Leaving this alone to see if it happens again. Will report the issue if it does.

[olukas]

It happened again (and again on IBM JDK 8): http://jenkins.hazelcast.com/job/jet-oss-master-ibm-jdk8/251/testReport/com.hazelcast.jet.elastic.impl/

[olukas]

Failed again with IBM JDK 8: http://jenkins.hazelcast.com/job/jet-oss-master-ibm-jdk8/257/testReport/com.hazelcast.jet.elastic.impl/

[frant-hartm]

The root cause of the exception slightly differs on the updated version of ibm jdk:

Caused by: com.sun.tools.attach.AttachNotSupportedException: target 147145 not found
	at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTargetImpl(OpenJ9VirtualMachine.java:150)
	at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.lambda$attachTarget$1(OpenJ9VirtualMachine.java:128)
	at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine$$Lambda$37/0000000078EFDAF0.run(Unknown Source)
	at java.security.AccessController.doPrivileged(AccessController.java:734)
	at com.ibm.tools.attach.attacher.OpenJ9VirtualMachine.attachTarget(OpenJ9VirtualMachine.java:130)
	at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:65)
	at com.ibm.tools.attach.attacher.OpenJ9AttachProvider.attachVirtualMachine(OpenJ9AttachProvider.java:48)
	at com.sun.tools.attach.VirtualMachine.attach(VirtualMachine.java:208)
	... 51 more