helpdesk icon indicating copy to clipboard operation
helpdesk copied to clipboard

Published 20 hours ago • verified publisher icon jenkins-infra

Intermittent out of memory for Java 21 builds of Jenkins core on ci.jenkins.io

Open MarkEWaite opened this issue 1 year ago • 7 comments
trafficstars

Service(s)

ci.jenkins.io

Summary

The Java 21 builds of Jenkins core on ci.jenkins.io have been reporting "out of memory" failures as in https://github.com/jenkinsci/jenkins/pull/8793/checks?check_run_id=19835902208 . It is not a consistent failure every time, but they have happened at least several times per week.

Reproduction steps

  1. Run a build of Jenkins core on ci.jenkins.io and see a Java 21 failure like:
    • https://ci.jenkins.io/job/Core/job/jenkins/job/master/5682/pipeline-graph/
    • https://ci.jenkins.io/job/Core/job/jenkins/job/master/5663/pipeline-graph/

MarkEWaite avatar Dec 20 '23 20:12 MarkEWaite

Adding the error message (in case the build is recycled):

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]

[1343](https://ci.jenkins.io/job/Core/job/jenkins/job/PR-8793/2/pipeline-console/?start-byte=1535&selected-node=210#log-1343)
19:54:18  org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception

Also (to search for memory metrics):

Running on [jnlp-maven-21-qgn7t](https://ci.jenkins.io/computer/jnlp%2Dmaven%2D21%2Dqgn7t/) in /home/jenkins/agent/workspace/Core_jenkins_PR-8793

dduportal avatar Dec 21 '23 07:12 dduportal

Different angles to diagnose:

  • Were the agent OOM-killed? If not, then check the memory usage (metrics)
  • Check if JDK21 has weird behavior inside container (limit/requests)
  • Check which cluster was used (is there a failing correlation with AWS or DOKS?)
  • Try to run on a VM agent instead of container to see if it disappear

dduportal avatar Jan 02 '24 15:01 dduportal

It is possible that the baseline memory requirements have increased beyond the 1.1 GB heap allocated to the Maven process in https://github.com/jenkinsci/jenkins/blob/95bff130c32c5c1fb049792d6419517748658633/.mvn/jvm.config. One way to prove this would be to set up a reproduction scenario (e.g., run the test in a loop and observe that it always fails within n iterations) and then slowly increase -Xmx until the build no longer fails within the same n iterations as in https://github.com/jenkinsci/jenkins/pull/6652. I am not sure how plausible this theory is, since I never see the failure locally (though I have 16 GB of RAM), only in CI. If the theory is disproven, the next step would be to enable -XX:HeapDumpOnOutOfMemoryError, update the CI job to capture the heap dump, reproduce the problem, and then do traditional heap dump analysis to see where the memory is going.

basil avatar Jan 02 '24 19:01 basil

I've started 4 draft pull requests to build with Java 21 on Linux. I'm also running a series of 16 builds of Jenkins core on my local computer with a command line that is very close to the original command line used in the Jenkins job.

MarkEWaite avatar Jan 09 '24 17:01 MarkEWaite

From https://ci.jenkins.io/job/Core/job/jenkins/job/master/5747/consoleFull

[ERROR] Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception: Java heap space -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.maven.plugins:maven-war-plugin:3.4.0:war (default-war) on project jenkins-war: Error assembling WAR: Problem creating war: Execution exception
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:333)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.apache.maven.plugin.MojoExecutionException: Error assembling WAR: Problem creating war: Execution exception
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:181)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: org.codehaus.plexus.archiver.ArchiverException: Problem creating war: Execution exception
	at org.codehaus.plexus.archiver.AbstractArchiver.createArchive (AbstractArchiver.java:1052)
	at org.apache.maven.archiver.MavenArchiver.createArchive (MavenArchiver.java:707)
	at org.apache.maven.plugins.war.WarMojo.performPackaging (WarMojo.java:237)
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:179)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.io.IOException: Execution exception
	at org.codehaus.plexus.archiver.zip.AbstractZipArchiver.close (AbstractZipArchiver.java:798)
	at org.codehaus.plexus.archiver.AbstractArchiver.createArchive (AbstractArchiver.java:1039)
	at org.apache.maven.archiver.MavenArchiver.createArchive (MavenArchiver.java:707)
	at org.apache.maven.plugins.war.WarMojo.performPackaging (WarMojo.java:237)
	at org.apache.maven.plugins.war.WarMojo.execute (WarMojo.java:179)
	at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:126)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute2 (MojoExecutor.java:328)
	at org.apache.maven.lifecycle.internal.MojoExecutor.doExecute (MojoExecutor.java:316)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:212)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:174)
	at org.apache.maven.lifecycle.internal.MojoExecutor.access$000 (MojoExecutor.java:75)
	at org.apache.maven.lifecycle.internal.MojoExecutor$1.run (MojoExecutor.java:162)
	at org.apache.maven.plugin.DefaultMojosExecutionStrategy.execute (DefaultMojosExecutionStrategy.java:39)
	at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:159)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:105)
	at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:73)
	at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:53)
	at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:118)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:261)
	at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:173)
	at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:101)
	at org.apache.maven.cli.MavenCli.execute (MavenCli.java:906)
	at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:283)
	at org.apache.maven.cli.MavenCli.main (MavenCli.java:206)
	at jdk.internal.reflect.DirectMethodHandleAccessor.invoke (DirectMethodHandleAccessor.java:103)
	at java.lang.reflect.Method.invoke (Method.java:580)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:283)
	at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:226)
	at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:407)
	at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:348)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.needNewBuffer (ByteArrayOutputStream.java:153)
	at org.codehaus.plexus.archiver.zip.ByteArrayOutputStream.write (ByteArrayOutputStream.java:192)
	at org.apache.commons.io.output.ThresholdingOutputStream.write (ThresholdingOutputStream.java:232)
	at org.codehaus.plexus.archiver.zip.DeferredScatterOutputStream.writeOut (DeferredScatterOutputStream.java:44)
	at org.apache.commons.compress.archivers.zip.StreamCompressor$ScatterGatherBackingStoreCompressor.writeOut (StreamCompressor.java:83)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeCounted (StreamCompressor.java:312)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate (StreamCompressor.java:203)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflateUntilInputIsNeeded (StreamCompressor.java:230)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.writeDeflated (StreamCompressor.java:322)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.write (StreamCompressor.java:299)
	at org.apache.commons.compress.archivers.zip.StreamCompressor.deflate (StreamCompressor.java:221)
	at org.apache.commons.compress.archivers.zip.ScatterZipOutputStream.addArchiveEntry (ScatterZipOutputStream.java:181)
	at org.apache.commons.compress.archivers.zip.ParallelScatterZipCreator.lambda$createCallable$1 (ParallelScatterZipCreator.java:220)
	at org.apache.commons.compress.archivers.zip.ParallelScatterZipCreator$$Lambda/0x00007fdab8f25ec8.call (Unknown Source)
	at java.util.concurrent.FutureTask.run (FutureTask.java:317)
	at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1144)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:642)
	at java.lang.Thread.runWith (Thread.java:1596)
	at java.lang.Thread.run (Thread.java:1583)

basil avatar Jan 10 '24 01:01 basil

Should be resolved by https://github.com/jenkinsci/jenkins/pull/8856, builds are looking more stable.

timja avatar Jan 13 '24 16:01 timja

Unfortunately, even with https://github.com/jenkinsci/jenkins/pull/8856 , we are still seeing intermittent failures like a recent dependabot pull request https://ci.jenkins.io/job/Core/job/jenkins/job/PR-8865/1/execution/node/271/log/#1318

MarkEWaite avatar Jan 14 '24 14:01 MarkEWaite

Unfortunately, even with jenkinsci/jenkins#8856 , we are still seeing intermittent failures like a recent dependabot pull request https://ci.jenkins.io/job/Core/job/jenkins/job/PR-8865/1/execution/node/271/log/#1318

Hi @MarkEWaite @timja @NotMyFault , could you confirm or infirm if these intermittent failures are still present (given the ci.jenkins.io infrastructure has been upgraded since the past 4 months)?

dduportal avatar Jun 11 '24 09:06 dduportal

As far as I can tell, there have not been out of memory failures on Java 21 builds in recent memory. I think this can be closed.

MarkEWaite avatar Jun 11 '24 11:06 MarkEWaite

Ack. I'm closing then, thanks!

dduportal avatar Jun 11 '24 12:06 dduportal