infrastructure icon indicating copy to clipboard operation
infrastructure copied to clipboard
verified publisher icon adoptium

OOM on AIX jdk_net_1 && jdk_net_0 tests (extended.openjdk)

Open RadekCap opened this issue 1 year ago • 9 comments

Test Info Test Name: jdk_net_1 Test Duration: 1 hr 45 min 17 sec Machine: test-osuosl-aix72-ppc64-1 TRSS link for the test output: https://trss.adoptium.net/output/test?id=65d35cd943ff67006e58d3c3

Build Info Build Name: Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_0 Jenkins Build start time: Feb 19 2024, 03:20 am Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_0/110/ TRSS link for the build: https://trss.adoptium.net/allTestsInfo?buildId=65d359fb43ff67006e589120

Java Version openjdk version "11.0.23-beta" 2024-04-16 OpenJDK Runtime Environment Temurin-11.0.23+3-202402190059 (build 11.0.23-beta+3-ea) OpenJDK 64-Bit Server VM Temurin-11.0.23+3-202402190059 (build 11.0.23-beta+3-ea, mixed mode)

This test has been failed 19 times since Apr 19 2023, 08:57 pm Java Version when the issue first seen openjdk version "11.0.19" 2023-04-18 OpenJDK Runtime Environment Temurin-11.0.19+7 (build 11.0.19+7) OpenJDK 64-Bit Server VM Temurin-11.0.19+7 (build 11.0.19+7, mixed mode) Jenkins Build URL: https://ci.adoptium.net/job/Test_openjdk11_hs_extended.openjdk_ppc64_aix_testList_0/76/

The test failed on machine test-osuosl-aix72-ppc64-1 3 times The test failed on machine test-osuosl-aix72-ppc64-4 3 times The test failed on machine test-osuosl-aix72-ppc64-5 3 times The test failed on machine test-osuosl-aix72-ppc64-3 5 times The test failed on machine test-osuosl-aix72-ppc64-2 4 times The test failed on machine test-osuosl-aix72-ppc64-6 1 times

Rerun in Grinder

From jdk_net, 2 testcases failing:

  • java/net/httpclient/SpecialHeadersTest.java
[2024-04-17T17:22:00.950Z] test SpecialHeadersTest.testAsync("http://localhost:60636/http1", "REFERER: upper", true): failure
[2024-04-17T17:22:00.950Z] java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
[2024-04-17T17:22:00.950Z] 	at java.base/java.lang.Thread.start0(Native Method)
[2024-04-17T17:22:00.950Z] 	at java.base/java.lang.Thread.start(Thread.java:798)
[2024-04-17T17:22:00.950Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
[2024-04-17T17:22:00.950Z] 	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
[2024-04-17T17:22:00.950Z] 	at java.base/java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2597)
[2024-04-17T17:22:00.950Z] 	at java.net.http/jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:246)
[2024-04-17T17:22:00.950Z] 	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:632)
[2024-04-17T17:22:00.950Z] 	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:586)
[2024-04-17T17:22:00.950Z] 	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:578)
[2024-04-17T17:22:00.950Z] 	at java.net.http/jdk.internal.net.http.HttpClientFacade.sendAsync(HttpClientFacade.java:129)
[2024-04-17T17:22:00.950Z] 	at SpecialHeadersTest.testAsync(SpecialHeadersTest.java:296)
[2024-04-17T17:22:00.950Z] 	at jdk.internal.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
[2024-04-17T17:22:00.950Z] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-04-17T17:22:00.951Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
[2024-04-17T17:22:00.951Z] 	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
[2024-04-17T17:22:00.951Z] 	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestRunner.privateRun(TestRunner.java:764)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestRunner.run(TestRunner.java:585)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
[2024-04-17T17:22:00.951Z] 	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestNG.runSuites(TestNG.java:1069)
[2024-04-17T17:22:00.951Z] 	at org.testng.TestNG.run(TestNG.java:1037)
[2024-04-17T17:22:00.951Z] 	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
[2024-04-17T17:22:00.951Z] 	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:54)
[2024-04-17T17:22:00.951Z] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[2024-04-17T17:22:00.951Z] 	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
[2024-04-17T17:22:00.951Z] 	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
[2024-04-17T17:22:00.951Z] 	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
[2024-04-17T17:22:00.951Z] 	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
[2024-04-17T17:22:00.951Z] 	at java.base/java.lang.Thread.run(Thread.java:829)
  • java/net/httpclient/StreamingBody.java
13:25:54  [55.838s][warning][os,thread] Failed to start thread "Unknown thread" - pthread_create failed (11=EAGAIN) for attributes: stacksize: 2112k, guardsize: 0k, detached.
13:25:54  [55.838s][warning][os,thread] Failed to start the native thread for java.lang.Thread "HttpClient-289-Worker-0"
13:25:54  test StreamingBody.test("http://localhost:61105/http2/streamingbody/y"): failure
13:25:54  java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached
13:25:54  	at java.base/java.lang.Thread.start0(Native Method)
13:25:54  	at java.base/java.lang.Thread.start(Thread.java:798)
13:25:54  	at java.base/java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:937)
13:25:54  	at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1354)
13:25:54  	at java.base/java.util.concurrent.CompletableFuture.completeAsync(CompletableFuture.java:2597)
13:25:54  	at java.net.http/jdk.internal.net.http.MultiExchange.responseAsync(MultiExchange.java:246)
13:25:54  	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:632)
13:25:54  	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:586)
13:25:54  	at java.net.http/jdk.internal.net.http.HttpClientImpl.sendAsync(HttpClientImpl.java:578)
13:25:54  	at java.net.http/jdk.internal.net.http.HttpClientFacade.sendAsync(HttpClientFacade.java:129)
13:25:54  	at StreamingBody.test(StreamingBody.java:103)
13:25:54  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
13:25:54  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
13:25:54  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:25:54  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
13:25:54  	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:132)
13:25:54  	at org.testng.internal.TestInvoker.invokeMethod(TestInvoker.java:599)
13:25:54  	at org.testng.internal.TestInvoker.invokeTestMethod(TestInvoker.java:174)
13:25:54  	at org.testng.internal.MethodRunner.runInSequence(MethodRunner.java:46)
13:25:54  	at org.testng.internal.TestInvoker$MethodInvocationAgent.invoke(TestInvoker.java:822)
13:25:54  	at org.testng.internal.TestInvoker.invokeTestMethods(TestInvoker.java:147)
13:25:54  	at org.testng.internal.TestMethodWorker.invokeTestMethods(TestMethodWorker.java:146)
13:25:54  	at org.testng.internal.TestMethodWorker.run(TestMethodWorker.java:128)
13:25:54  	at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
13:25:54  	at org.testng.TestRunner.privateRun(TestRunner.java:764)
13:25:54  	at org.testng.TestRunner.run(TestRunner.java:585)
13:25:54  	at org.testng.SuiteRunner.runTest(SuiteRunner.java:384)
13:25:54  	at org.testng.SuiteRunner.runSequentially(SuiteRunner.java:378)
13:25:54  	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:337)
13:25:54  	at org.testng.SuiteRunner.run(SuiteRunner.java:286)
13:25:54  	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:53)
13:25:54  	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:96)
13:25:54  	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1218)
13:25:54  	at org.testng.TestNG.runSuitesLocally(TestNG.java:1140)
13:25:54  	at org.testng.TestNG.runSuites(TestNG.java:1069)
13:25:54  	at org.testng.TestNG.run(TestNG.java:1037)
13:25:54  	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:94)
13:25:54  	at com.sun.javatest.regtest.agent.TestNGRunner.main(TestNGRunner.java:54)
13:25:54  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
13:25:54  	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
13:25:54  	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
13:25:54  	at java.base/java.lang.reflect.Method.invoke(Method.java:566)
13:25:54  	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
13:25:54  	at java.base/java.lang.Thread.run(Thread.java:829)

RadekCap avatar Apr 19 '24 08:04 RadekCap

The deep history indicates it's a pure failure: https://trss.adoptium.net/deepHistory?testId=662075f3879917006ea74ab6

RadekCap avatar Apr 19 '24 08:04 RadekCap

Both ends on network connection failures:

INFO: ERROR: java.io.IOException: A connection with a remote socket was reset by that socket.

and

INFO: MISC: Closing: PlainHttpConnection: HttpConnection: java.nio.channels.SocketChannel[connected local=/127.0.0.1:61438 remote=localhost/127.0.0.1:61106]
TestServer: Connection writer stopping
Apr 17, 2024 5:25:47 PM jdk.internal.net.http.PlainHttpConnection close
INFO: MISC: Closing: PlainHttpConnection: HttpConnection: java.nio.channels.SocketChannel[connected local=/127.0.0.1:61437 remote=localhost/127.0.0.1:61106]

I'm attaching jtr files. SpecialHeadersTest.jtr.txt

SpecialHeadersTest.jtr.txt

RadekCap avatar Apr 19 '24 08:04 RadekCap

Updated names as jdk_net_0 has the same failures.

RadekCap avatar Apr 19 '24 09:04 RadekCap

The two testcases are failing with OutOfMemoryError, which could be a limitation of the machines we have on the public Jenkins server. Trying a run on the temurin-compliance Jenkins server to see if same issue occurs (for those with access to that private server, the link is TCGrinder/4238) - passes on jck-skytap-aix72-ppc64-4 Grinder_20240419103418_JDK11_AIX.tap.txt

I will transfer this issue to the infrastructure repository to see if there is a way to ensure we have same capacity / config on the public AIX machines versus the one attached to the TC Jenkins server.

smlambert avatar Apr 19 '24 10:04 smlambert

I've changed this issue title to be a generic limits issue for AIX. @andrew-m-leonard is this the same as what you saw at some point in the last week? It mentions java.lang.OutOfMemoryError: unable to create native thread: which sounds similar to what you were seeing.

Also noting that ref https://github.com/adoptium/infrastructure/issues/3065#issuecomment-2493619252 there is an error Execution failed: main threw exception: java.lang.OutOfMemoryError: Unable to allocate 1073741824 bytes occuring in java/nio/channels/FileChannel/LargeGatheringWrite.java (I'm expecting to close that issue when https://github.com/adoptium/aqa-tests/pull/5771 is merged, which will mean this issue can be used to track that too.

Based on earlier comment I'm also trying that on the TC server with grinders 4639-4641 (edit: All failed with TEST RESULT: Failed. Execution failed: 'main' threw exception: java.io.IOException: No space left on device since the test tries to write ~2GiB to /tmp location (Ref: https://github.com/adoptium/infrastructure/issues/3129) and the TC machines don't have enough available.

sxa avatar Nov 22 '24 15:11 sxa

I've changed this issue title to be a generic limits issue for AIX. @andrew-m-leonard is this the same as what you saw at some point in the last week? It mentions java.lang.OutOfMemoryError: unable to create native thread: which sounds similar to what you were seeing.

Also noting that ref #3065 (comment) there is an error Execution failed: main threw exception: java.lang.OutOfMemoryError: Unable to allocate 1073741824 bytes occuring in java/nio/channels/FileChannel/LargeGatheringWrite.java (I'm expecting to close that issue when adoptium/aqa-tests#5771 is merged, which will mean this issue can be used to track that too.

Based on earlier comment I'm also trying that on the TC server with grinders 4639-4641 (edit: All failed with TEST RESULT: Failed. Execution failed: 'main' threw exception: java.io.IOException: No space left on device since the test tries to write ~2GiB to /tmp location (Ref: #3129) and the TC machines don't have enough available.

The only issue I saw last week I think was the timeout after 5mins scheduling the nodes (I think!)

andrew-m-leonard avatar Nov 25 '24 09:11 andrew-m-leonard

The only issue I saw last week I think was the timeout after 5mins scheduling the nodes (I think!)

~~Didn't you have something related to memory/threads too?~~ Ah no it was Scott in 3513

sxa avatar Nov 28 '24 16:11 sxa

We're still seeing this issue in July 2025. Stewart's proposed solution is an increased LDR_CNTRL=MAXDATA size. Haroon has kindly volunteered to make this change to the jenkins config of the AIX machines.

adamfarley avatar Jul 22 '25 15:07 adamfarley

Discovered today that this works if you increase the value of LDR_CNTRL to MAXDATA=0x0A0000000 from the normal MAXDATA=0x080000000 then it seems to pass (based on testing at the command line). This is configured in the jenkins agent definition. First time I've seen something need a little more than 8

sxa avatar Jul 22 '25 15:07 sxa