Get I/O exception during sandboxed execution (File exists) when running bazel test.

[cyrilhuang]

Description of the bug:

When running bazel test executing in sandbox, we get the error I/O exception during sandboxed execution: /xxxx/.cache/bazel/_bazel_user/553e729b1be2c00c3778349b7d539a93/sandbox/processwrapper-sandbox/7276/execroot/project/external/bazel_tools/tools/test/generate-xml.sh (File exists) for one test.

Seems we cannot reproduce it after rerun the cmd. And we have no idea on the error log.

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

No response

Which operating system are you running Bazel on?

Ubuntu

What is the output of bazel info release?

release 4.2.2

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

No response

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

No response

Have you found anything relevant by searching the web?

https://github.com/bazelbuild/rules_pkg/issues/135 This seems to be related, but not exactly the same.

Any other information, logs, or outputs that you want to share?

com.google.devtools.build.lib.skyframe.ActionExecutionFunction$ActionExecutionFunctionException: com.google.devtools.build.lib.actions.AlreadyReportedActionExecutionException: I/O exception during sandboxed execution: /xxxx/.cache/bazel/_bazel_user/553e729b1be2c00c3778349b7d539a93/sandbox/processwrapper-sandbox/7276/execroot/project/external/bazel_tools/tools/test/generate-xml.sh (File exists)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:326)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:477)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.devtools.build.lib.actions.AlreadyReportedActionExecutionException: I/O exception during sandboxed execution: /xxxx/.cache/bazel/_bazel_user/553e729b1be2c00c3778349b7d539a93/sandbox/processwrapper-sandbox/7276/execroot/project/external/bazel_tools/tools/test/generate-xml.sh (File exists)
        ... 6 more
Caused by: com.google.devtools.build.lib.actions.UserExecException: I/O exception during sandboxed execution: /xxxx/.cache/bazel/_bazel_user/553e729b1be2c00c3778349b7d539a93/sandbox/processwrapper-sandbox/7276/execroot/project/external/bazel_tools/tools/test/generate-xml.sh (File exists)
        at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:95)
        at com.google.devtools.build.lib.sandbox.SandboxModule$SandboxFallbackSpawnRunner.exec(SandboxModule.java:489)
        at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:238)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:144)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:106)
        at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
        at com.google.devtools.build.lib.exec.StandaloneTestStrategy.beginTestAttempt(StandaloneTestStrategy.java:426)
        at com.google.devtools.build.lib.exec.StandaloneTestStrategy.access$200(StandaloneTestStrategy.java:82)
        at com.google.devtools.build.lib.exec.StandaloneTestStrategy$StandaloneTestRunnerSpawn.beginExecution(StandaloneTestStrategy.java:652)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.beginIfNotCancelled(TestRunnerAction.java:880)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.beginExecution(TestRunnerAction.java:847)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:905)
        at com.google.devtools.build.lib.analysis.test.TestRunnerAction.execute(TestRunnerAction.java:896)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$5.execute(SkyframeActionExecutor.java:855)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:1016)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:975)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:129)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:81)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:472)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:834)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:307)
        ... 5 more
Caused by: java.io.IOException: /xxxx/.cache/bazel/_bazel_user/553e729b1be2c00c3778349b7d539a93/sandbox/processwrapper-sandbox/7276/execroot/project/external/bazel_tools/tools/test/generate-xml.sh (File exists)
        at com.google.devtools.build.lib.unix.NativePosixFiles.symlink(Native Method)
        at com.google.devtools.build.lib.unix.UnixFileSystem.createSymbolicLink(UnixFileSystem.java:336)
        at com.google.devtools.build.lib.vfs.Path.createSymbolicLink(Path.java:561)
        at com.google.devtools.build.lib.sandbox.SymlinkedSandboxedSpawn.copyFile(SymlinkedSandboxedSpawn.java:57)
        at com.google.devtools.build.lib.sandbox.AbstractContainerizingSandboxedSpawn.createInputs(AbstractContainerizingSandboxedSpawn.java:152)
        at com.google.devtools.build.lib.sandbox.AbstractContainerizingSandboxedSpawn.createFileSystem(AbstractContainerizingSandboxedSpawn.java:93)
        at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.runSpawn(AbstractSandboxSpawnRunner.java:117)
        at com.google.devtools.build.lib.sandbox.AbstractSandboxSpawnRunner.exec(AbstractSandboxSpawnRunner.java:90)
        ... 26 more
230110 05:15:13.794:I 31 [com.google.devtools.build.lib.buildtool.ExecutionTool.saveActionCache] Spent 0 milliseconds doing Saving action cache

[sgowroji]

Hi @cyrilhuang, Please provide minimal steps to reproduce the above issue. It will be hard to troubleshoot.Thanks!

[cyrilhuang]

Hi @cyrilhuang, Please provide minimal steps to reproduce the above issue. It will be hard to troubleshoot.Thanks!

We still do not have the minimal steps yet. We only encounter this once, and running on another try, it cannot be reproduced.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 30 days. It will be closed in the next 7 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler".

[joeljeske]

I'm experiencing the same on 6.0.0

[joeleba]

@joeljeske do you have a way to reproduce this issue? Thanks.

[joeljeske]

No I do not. It’s very intermittent.

[github-actions[bot]]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[github-actions[bot]]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!