Podman randomly fails with java.io.IOException: Broken pipe

[MarkRx]

The gradle docker plugin with podman-system-service may randomly fail with a broken pipe error. I believe this is because the caching the docker client will not support the ephemeral podman-system-service with a default expiration of 5 seconds. If enough time passes between podman calls the client will attempt to use a closed connection resulting in a broken pipe error.

From the podman-system-service documentation:

--time, -t The time until the session expires in seconds. The default is 5 seconds. A value of 0 means no timeout, therefore the session does not expire.

The default timeout can be changed via the service_timeout=VALUE field in containers.conf. See containers.conf(5) for more information.

Expected Behavior

One of the following is an option:

1. Provide ability to disable client caching
2. Document that when using podman-system-service the session timeout should be increased or disabled

Current Behavior

The gradle docker plugin may fail with a broken pipe error if enough time passes between docker/podman commands

Steps to Reproduce (for bugs)

1. Setup podman and a systemd podman service
2. Set DOCKER_HOST environment variable to use podman socket
3. Put a sleep between two docker steps. Example:

def myImage = . . .

tasks.register('dockerBuild', DockerBuildImage) {
    inputDir = file("docker")
    images = [ myImage ]
}

tasks.register('sleep', DefaultTask) {
    dependsOn 'dockerBuild'
    doLast {
        println("Sleeping for 10 seconds")
        Thread.sleep(10000)
    }
}

tasks.register('dockerPush', DockerPushImage) {
    images = [ myImage ]
    dependsOn 'sleep'
}

Your Environment

RHEL 9 podman 4.6.1 gradle-docker-plugin 9.4.0 JDK 17

Stacktrace

org.gradle.api.tasks.TaskExecutionException: Execution failed for task ':dockerPush'. at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.lambda$executeIfValid$1(ExecuteActionsTaskExecuter.java:148) at org.gradle.internal.Try$Failure.ifSuccessfulOrElse(Try.java:282) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.executeIfValid(ExecuteActionsTaskExecuter.java:146) at org.gradle.api.internal.tasks.execution.ExecuteActionsTaskExecuter.execute(ExecuteActionsTaskExecuter.java:134) at org.gradle.api.internal.tasks.execution.FinalizePropertiesTaskExecuter.execute(FinalizePropertiesTaskExecuter.java:46) at org.gradle.api.internal.tasks.execution.ResolveTaskExecutionModeExecuter.execute(ResolveTaskExecutionModeExecuter.java:51) at org.gradle.api.internal.tasks.execution.SkipTaskWithNoActionsExecuter.execute(SkipTaskWithNoActionsExecuter.java:57) at org.gradle.api.internal.tasks.execution.SkipOnlyIfTaskExecuter.execute(SkipOnlyIfTaskExecuter.java:74) at org.gradle.api.internal.tasks.execution.CatchExceptionTaskExecuter.execute(CatchExceptionTaskExecuter.java:36) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.executeTask(EventFiringTaskExecuter.java:77) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:55) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter$1.call(EventFiringTaskExecuter.java:52) at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204) at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199) at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66) at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59) at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59) at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53) at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73) at org.gradle.api.internal.tasks.execution.EventFiringTaskExecuter.execute(EventFiringTaskExecuter.java:52) at org.gradle.execution.plan.LocalTaskNodeExecutor.execute(LocalTaskNodeExecutor.java:42) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:331) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$InvokeNodeExecutorsAction.execute(DefaultTaskExecutionGraph.java:318) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.lambda$execute$0(DefaultTaskExecutionGraph.java:314) at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:314) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph$BuildOperationAwareExecutionAction.execute(DefaultTaskExecutionGraph.java:303) at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.execute(DefaultPlanExecutor.java:463) at org.gradle.execution.plan.DefaultPlanExecutor$ExecutorWorker.run(DefaultPlanExecutor.java:380) at org.gradle.execution.plan.DefaultPlanExecutor.process(DefaultPlanExecutor.java:116) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph.executeWithServices(DefaultTaskExecutionGraph.java:138) at org.gradle.execution.taskgraph.DefaultTaskExecutionGraph.execute(DefaultTaskExecutionGraph.java:123) at org.gradle.execution.SelectedTaskExecutionAction.execute(SelectedTaskExecutionAction.java:35) at org.gradle.execution.DryRunBuildExecutionAction.execute(DryRunBuildExecutionAction.java:51) at org.gradle.execution.BuildOperationFiringBuildWorkerExecutor$ExecuteTasks.call(BuildOperationFiringBuildWorkerExecutor.java:54) at org.gradle.execution.BuildOperationFiringBuildWorkerExecutor$ExecuteTasks.call(BuildOperationFiringBuildWorkerExecutor.java:43) at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:204) at org.gradle.internal.operations.DefaultBuildOperationRunner$CallableBuildOperationWorker.execute(DefaultBuildOperationRunner.java:199) at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:66) at org.gradle.internal.operations.DefaultBuildOperationRunner$2.execute(DefaultBuildOperationRunner.java:59) at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:157) at org.gradle.internal.operations.DefaultBuildOperationRunner.execute(DefaultBuildOperationRunner.java:59) at org.gradle.internal.operations.DefaultBuildOperationRunner.call(DefaultBuildOperationRunner.java:53) at org.gradle.internal.operations.DefaultBuildOperationExecutor.call(DefaultBuildOperationExecutor.java:73) at org.gradle.execution.BuildOperationFiringBuildWorkerExecutor.execute(BuildOperationFiringBuildWorkerExecutor.java:40) at org.gradle.internal.build.DefaultBuildLifecycleController.lambda$executeTasks$10(DefaultBuildLifecycleController.java:313) at org.gradle.internal.model.StateTransitionController.doTransition(StateTransitionController.java:266) at org.gradle.internal.model.StateTransitionController.lambda$tryTransition$8(StateTransitionController.java:177) at org.gradle.internal.work.DefaultSynchronizer.withLock(DefaultSynchronizer.java:44) at org.gradle.internal.model.StateTransitionController.tryTransition(StateTransitionController.java:177) at org.gradle.internal.build.DefaultBuildLifecycleController.executeTasks(DefaultBuildLifecycleController.java:304) at org.gradle.internal.build.DefaultBuildWorkGraphController$DefaultBuildWorkGraph.runWork(DefaultBuildWorkGraphController.java:220) at org.gradle.internal.work.DefaultWorkerLeaseService.withLocks(DefaultWorkerLeaseService.java:264) at org.gradle.internal.work.DefaultWorkerLeaseService.runAsWorkerThread(DefaultWorkerLeaseService.java:128) at org.gradle.composite.internal.DefaultBuildController.doRun(DefaultBuildController.java:181) at org.gradle.composite.internal.DefaultBuildController.access$000(DefaultBuildController.java:50) at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.lambda$run$0(DefaultBuildController.java:198) at org.gradle.internal.operations.CurrentBuildOperationRef.with(CurrentBuildOperationRef.java:80) at org.gradle.composite.internal.DefaultBuildController$BuildOpRunnable.run(DefaultBuildController.java:198) at org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:64) at org.gradle.internal.concurrent.AbstractManagedExecutor$1.run(AbstractManagedExecutor.java:47) Caused by: java.lang.RuntimeException: java.io.IOException: Broken pipe at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:195) at com.github.dockerjava.httpclient5.ApacheDockerHttpClient.execute(ApacheDockerHttpClient.java:9) at com.github.dockerjava.core.DefaultInvocationBuilder.execute(DefaultInvocationBuilder.java:228) at com.github.dockerjava.core.DefaultInvocationBuilder.lambda$executeAndStream$1(DefaultInvocationBuilder.java:269) Caused by: java.io.IOException: Broken pipe at com.github.dockerjava.transport.UnixSocket$WrappedWritableByteChannel.write(UnixSocket.java:95) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flushBuffer(SessionOutputBufferImpl.java:117) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.SessionOutputBufferImpl.flush(SessionOutputBufferImpl.java:126) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.BHttpConnectionBase.flush(BHttpConnectionBase.java:309) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.DefaultBHttpClientConnection.flush(DefaultBHttpClientConnection.java:61) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:144) at com.github.dockerjava.httpclient5.HijackingHttpRequestExecutor.execute(HijackingHttpRequestExecutor.java:50) at com.bmuschko.gradle.docker.shaded.org.apache.hc.core5.http.impl.io.HttpRequestExecutor.execute(HttpRequestExecutor.java:218) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.io.PoolingHttpClientConnectionManager$InternalConnectionEndpoint.execute(PoolingHttpClientConnectionManager.java:596) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.InternalExecRuntime.execute(InternalExecRuntime.java:215) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.MainClientExec.execute(MainClientExec.java:107) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ConnectExec.execute(ConnectExec.java:181) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ProtocolExec.execute(ProtocolExec.java:172) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.HttpRequestRetryExec.execute(HttpRequestRetryExec.java:93) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ContentCompressionExec.execute(ContentCompressionExec.java:128) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement$1.proceed(ExecChainElement.java:57) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.RedirectExec.execute(RedirectExec.java:116) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.ExecChainElement.execute(ExecChainElement.java:51) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.InternalHttpClient.doExecute(InternalHttpClient.java:178) at com.bmuschko.gradle.docker.shaded.org.apache.hc.client5.http.impl.classic.CloseableHttpClient.execute(CloseableHttpClient.java:67) at com.github.dockerjava.httpclient5.ApacheDockerHttpClientImpl.execute(ApacheDockerHttpClientImpl.java:191)

[MarkRx]

As a workaround we disabled the session timeout.

Documentation

[attiand]

We had the same problem in our CI-environment. The problem was that the gradle cache directory grow large and was placed in the workspace and hence the build context. Adding the cache directory to .dockerignore fixed our problem.