FAILED_PRECONDITION The directory `/` was not found in the CAS..

[lixin-wei]

I met this error when using buildfarm, what does it mean? Could anyone help me?

[admin@lianxin-test lianxin-test.inc.alipay.net /home/admin/lianxin.wlx/X/ray]                                                                        [72/1915]
$ bazel build ray_pkg -c dbg --remote_executor=grpc://ray.buildfarm-0.inc.alipay.net:8980 --jobs=32 -k
INFO: Invocation ID: 6f4d0e59-3e87-41f9-bf44-065a90dd4e71
DEBUG: /home/admin/lianxin.wlx/X/ray/bazel/ray_deps_setup.bzl:78:14: No implicit mirrors used because urls were explicitly provided, name = com_google_protobuf
DEBUG: /home/admin/lianxin.wlx/X/ray/bazel/ray_deps_setup.bzl:78:14: No implicit mirrors used because urls were explicitly provided, name = com_github_spdlog
DEBUG: /home/admin/lianxin.wlx/X/ray/bazel/ray_deps_setup.bzl:78:14: No implicit mirrors used because urls were explicitly provided, name = io_opencensus_proto
WARNING: /home/admin/.cache/bazel/_bazel_admin/65c94b62544ddd3fd6adce5fe1096d8b/external/com_github_brpc_brpc/BUILD:458:19: in cc_library rule @com_github_brpc_brpc//:cc_brpc_idl_options_proto: target '@com_github_brpc_brpc//:cc_brpc_idl_options_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_proto
s': Only for backward compatibility. Do not use.
WARNING: /home/admin/.cache/bazel/_bazel_admin/65c94b62544ddd3fd6adce5fe1096d8b/external/com_github_brpc_brpc/BUILD:469:19: in cc_library rule @com_github_brpc
_brpc//:cc_brpc_internal_proto: target '@com_github_brpc_brpc//:cc_brpc_internal_proto' depends on deprecated target '@com_google_protobuf//:cc_wkt_protos': On
ly for backward compatibility. Do not use.
INFO: Analyzed target //:ray_pkg (0 packages loaded, 0 targets configured).
INFO: Found 1 target...
ERROR: /home/admin/lianxin.wlx/X/ray/BUILD.bazel:1833:12: Couldn't build file python/ray/_raylet.so: Linking of rule '//:python/ray/_raylet.so' failed (Exit 34
): java.io.IOException: com.google.devtools.build.lib.remote.ExecutionStatusException: FAILED_PRECONDITION: Action 031d059a5c2294e53cc3fc7240e9da552bfd9061d8e1
5681b377e8ea5eefd00d/142 is invalid: The directory `/` was not found in the CAS..
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:192)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$0(RemoteSpawnRunner.java:329)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:309)
        at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:241)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:132)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:100)
        at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)                                                      at com.google.devtools.build.lib.rules.cpp.CppLinkAction.beginExecution(CppLinkAction.java:332)                                                                at com.google.devtools.build.lib.actions.Action.execute(Action.java:124)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:785)                                                    at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:948)                                  at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:907)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:128)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:80)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:425)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:926)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:310)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:438)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.devtools.build.lib.remote.ExecutionStatusException: FAILED_PRECONDITION: Action 031d059a5c2294e53cc3fc7240e9da552bfd9061d8e15681b377e8ea$
eefd00d/142 is invalid: The directory `/` was not found in the CAS..
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.handleStatus(GrpcRemoteExecutor.java:69)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.getOperationResponse(GrpcRemoteExecutor.java:81)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$0(GrpcRemoteExecutor.java:155)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:134)
        ... 24 more                                                                                                                                            . Note: Remote connection/protocol failed with: execution failed java.io.IOException: com.google.devtools.build.lib.remote.ExecutionStatusException: FAILED_PR$CONDITION: Action 031d059a5c2294e53cc3fc7240e9da552bfd9061d8e15681b377e8ea5eefd00d/142 is invalid: The directory `/` was not found in the CAS..
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:192)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$0(RemoteSpawnRunner.java:329)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
        at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:309)
        at com.google.devtools.build.lib.exec.SpawnRunner.execAsync(SpawnRunner.java:241)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:132)
        at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:100)
        at com.google.devtools.build.lib.actions.SpawnStrategy.beginExecution(SpawnStrategy.java:47)
        at com.google.devtools.build.lib.exec.SpawnStrategyResolver.beginExecution(SpawnStrategyResolver.java:65)
        at com.google.devtools.build.lib.rules.cpp.CppLinkAction.beginExecution(CppLinkAction.java:332)
        at com.google.devtools.build.lib.actions.Action.execute(Action.java:124)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$4.execute(SkyframeActionExecutor.java:785)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.continueAction(SkyframeActionExecutor.java:948)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:907)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:128)
        at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:80)
        at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:425)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:926)
        at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:310)
        at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:438)
        at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:398)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
Caused by: com.google.devtools.build.lib.remote.ExecutionStatusException: FAILED_PRECONDITION: Action 031d059a5c2294e53cc3fc7240e9da552bfd9061d8e15681b377e8ea5
eefd00d/142 is invalid: The directory `/` was not found in the CAS..
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.handleStatus(GrpcRemoteExecutor.java:69)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.getOperationResponse(GrpcRemoteExecutor.java:81)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$0(GrpcRemoteExecutor.java:155)
        at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:237)
        at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:115)
        at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:134)
        ... 24 more
Target //:ray_pkg failed to build
INFO: Elapsed time: 203.317s, Critical Path: 202.51s
INFO: 88 processes: 18 remote cache hit, 5 local, 65 remote.
FAILED: Build did NOT complete successfully

[lixin-wei]

A wired thing is, when I run this action again, it will pass. This error only occurs when I build this action for the first time.

[werkt]

The validation procedure for your execution is attempting to observe the entire input tree of directories and ensure that they are present before scheduling the execution on a worker. It cannot find the root directory of the tree, indicated by the / path, and so errors the action.

Getting a different result when you rerun it is not a good indicator of deterministic behavior.

Regardless, because this error emits the action id, you can investigate using bf-cat to see what inputs the action 031d059a5c2294e53cc3fc7240e9da552bfd9061d8e15681b377e8ea5eefd00d/142 specifies: I expect that it will be a completely empty tree in some form or another, which is another good indication that something has gone wrong, and perhaps things are being done out of order.

Your output from bazel is also interesting: what version are you running? I introduced changes some time ago to provide much better reporting on the nature of FAILED_PRECONDITION contents that would have further illustrated the blob id for the directory in question, and avoided further feedback required here.

[lixin-wei]

Your output from bazel is also interesting: what version are you running?

3.4.1

you can investigate using bf-cat to see what inputs the action

I see, thx. I'll run it when I meet it again. I can't reproduce it now :(