infrastructure icon indicating copy to clipboard operation
infrastructure copied to clipboard
verified publisher icon adoptium

[jdk8 - windows64]: some java_util tests failed with `insufficient memory`

Open sophia-guo opened this issue 3 years ago • 20 comments

https://ci.adoptopenjdk.net/job/Test_openjdk8_hs_sanity.openjdk_x86-64_windows/648/#showFailuresLink

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java.SerializedLambdaTest java/util/stream/test/org/openjdk/tests/java/util/FillableStringTest.java.FillableStringTest java/util/stream/test/org/openjdk/tests/java/util/stream/ConcatOpTest.java.ConcatOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/DistinctOpTest.java.DistinctOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/DoublePrimitiveOpsTests.java.DoublePrimitiveOpsTests java/util/stream/test/org/openjdk/tests/java/util/stream/FindAnyOpTest.java.FindAnyOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/FlatMapOpTest.java.FlatMapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/FlatMapOpTest.java.FlatMapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java.InfiniteStreamWithLimitOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/InfiniteStreamWithLimitOpTest.java.InfiniteStreamWithLimitOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/MapOpTest.java.MapOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SequentialOpTest.java.SequentialOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SliceOpTest.java.SliceOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SliceOpTest.java.SliceOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SortedOpTest.java.SortedOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/SpliteratorTest.java.SpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamBuilderTest.java.StreamBuilderTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java.StreamSpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java.StreamSpliteratorTest java/util/stream/test/org/openjdk/tests/java/util/stream/TabulatorsTest.java.TabulatorsTest java/util/stream/test/org/openjdk/tests/java/util/stream/TabulatorsTest.java.TabulatorsTest java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java.TeeOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java.TeeOpTest java/util/stream/test/org/openjdk/tests/java/util/stream/ToArrayOpTest.java.ToArrayOpTest

09:03:34  STDOUT:
09:03:34  #
09:03:34  # There is insufficient memory for the Java Runtime Environment to continue.
09:03:34  # Native memory allocation (malloc) failed to allocate 1048576 bytes for AllocateHeap
09:03:34  # An error report file with more information is saved as:
09:03:34  # E:\workspace\Test_openjdk8_hs_sanity.openjdk_x86-64_windows\aqa-tests\TKG\output_16567659847658\jdk_util_1\work\java\util\stream\test\org\openjdk\tests\java\util\stream\TabulatorsTest\hs_err_pid19676.log

Failed on test-azure-win2012r2-x64-3 test-ibmcloud-win2012r2-x64-1.

sophia-guo avatar Jul 07 '22 17:07 sophia-guo

Rerunning

Problem machines: test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5143/console

test-ibmcloud-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5144/console

Other machines: test-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5145/console

test-azure-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5146/console

build-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5147/console

test-azure-win2016-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5148/console

test-azure-win2019-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5149/console

build-azure-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5150/console

build-alibaba-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5151/console

Haroon-Khel avatar Jul 08 '22 11:07 Haroon-Khel

Util tests look good. The insufficient memory error seems intermittent. The lang test did not run with the above grinders, i'll re run that one now

Haroon-Khel avatar Jul 08 '22 12:07 Haroon-Khel

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java

Problem machines: test-azure-win2012r2-x64-3 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5153/console

test-ibmcloud-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5154/console

Other machines: test-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5145/console

test-azure-win2012r2-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5155/console

build-ibmcloud-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5156/console

test-azure-win2016-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5157/console

test-azure-win2019-x64-1 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5158/console

build-azure-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5159/console

build-alibaba-win2012r2-x64-2 https://ci.adoptopenjdk.net/view/Test_grinder/job/Grinder/5160/console

Haroon-Khel avatar Jul 08 '22 12:07 Haroon-Khel

java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java looks good too. More evidence that this error is intermittent

Haroon-Khel avatar Jul 14 '22 16:07 Haroon-Khel

Have they failed at all in your recent runs? May be worth trying again on the machines it's failed on but running with 50 iterations or something like that and see if any of them fail :-)

sxa avatar Jul 15 '22 10:07 sxa

This needs analysis to understand if it's an intermittent failure or not. Bumping to October

sxa avatar Oct 03 '22 13:10 sxa

@sophia-guo @Haroon-Khel Has this been seen recently?

sxa avatar Apr 06 '23 10:04 sxa

https://ci.adoptium.net/view/Test_grinder/job/Grinder/7120/console jdk_util on test-azure-win2012r2-x64-3 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7121/console java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java on test-azure-win2012r2-x64-3

Haroon-Khel avatar Apr 11 '23 13:04 Haroon-Khel

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is not able to connect to jenkins. I can rdp into the machine. From there it looks like its having trouble accessing the internet, for example it cant connect to ci.adoptium.net in a web browser. @AdamBrousseau Are you able to see if test-ibmcloud-win2012r2-x64-1 is having network issues?

Haroon-Khel avatar Apr 11 '23 13:04 Haroon-Khel

https://ci.adoptium.net/view/Test_grinder/job/Grinder/7120/console jdk_util on test-azure-win2012r2-x64-3 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7121/console java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java on test-azure-win2012r2-x64-3

Both passed. Just test-ibmcloud-win2012r2-x64-1 left

Haroon-Khel avatar Apr 11 '23 14:04 Haroon-Khel

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is not able to connect to jenkins. I can rdp into the machine. From there it looks like its having trouble accessing the internet, for example it cant connect to ci.adoptium.net in a web browser. @AdamBrousseau Are you able to see if test-ibmcloud-win2012r2-x64-1 is having network issues?

Can it do anything e.g. ssh out? Download something via curl/wget? What errors are we getting on the machine? Does it seem to be hostname resolution for example?

(Clearly the networking isn't fundamentally broken as we can get into it ;-) )

sxa avatar Apr 11 '23 14:04 sxa

curl and wget did not work as it looks like it cannot resolve hostnames, website were not loading in a browser

image image

Haroon-Khel avatar Apr 12 '23 15:04 Haroon-Khel

https://ci.adoptium.net/computer/test-ibmcloud-win2012r2-x64-1/ is back online after manually giving it a dns server (8.8.8.8)

Haroon-Khel avatar Apr 14 '23 15:04 Haroon-Khel

jdk_util on test-ibmcloud-1 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7137/console No need for a separate java/util/stream/test/org/openjdk/tests/java/lang/invoke/SerializedLambdaTest.java grinder as it runs in jdk_util

Haroon-Khel avatar Apr 14 '23 15:04 Haroon-Khel

Out of the 10 iterations in the grinder, these tests were the common failures. Others were intermittent

Failed test cases: 
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamLinkTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/StreamSpliteratorTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/TeeOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/stream/ToArrayOpTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/FillableStringTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/MapTest.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/NullArgsTestCase.java
TEST: java/util/stream/test/org/openjdk/tests/java/util/SplittableRandomTest.java
Test results: passed: 662; failed: 8

Haroon-Khel avatar Apr 18 '23 10:04 Haroon-Khel

rebuilds

jdk11 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7138/console jdk 17 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7139/console jdk 20 https://ci.adoptium.net/view/Test_grinder/job/Grinder/7140/console

Haroon-Khel avatar Apr 18 '23 10:04 Haroon-Khel

Tests pass for jdk 11 17 and 20

Haroon-Khel avatar Apr 18 '23 11:04 Haroon-Khel

Heya Haroon. Do you think this could be related?

adamfarley avatar May 30 '23 13:05 adamfarley

Needs to be rerun on newer machines, alot (if not all) of the original machines are out of commission

Haroon-Khel avatar Apr 05 '24 15:04 Haroon-Khel

Are you suggesting that it would only fail now on win2012 machines or are you using another definition of "newer" there?

sxa avatar Apr 05 '24 17:04 sxa