nativelink
nativelink copied to clipboard
Published
20 hours ago •
TraceMachina
TraceMachina
K8s examples break with long running builds
When trying to build LLVM with the K8s setup, the connection drops after ~6000 targets. It appears that this is not recoverable, i.e. rerunning the build doesn't work, likely because the default cas sizes are too small. We should adjust the values to make the setup more robust for a "naive import". Another cause could be the Http2 gateway configuration which we should be switched to grpc anyways.
See:
- https://github.com/eomii/rules_ll/pull/213
ERROR: /home/aaron/.cache/bazel/_bazel_aaron/7532c8de286b66ea1aaee4ff51952e7c/external/llvm-project-overlay~~llvm_project_overlay~llvm-project/clang/BUILD.bazel:1003:11: Comp
iling clang/lib/Sema/SemaTemplate.cpp failed: (Exit 34): UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:241)
at com.google.devtools.build.lib.remote.RemoteExecutionService.executeRemotely(RemoteExecutionService.java:1516)
at com.google.devtools.build.lib.remote.RemoteSpawnRunner.lambda$exec$2(RemoteSpawnRunner.java:293)
at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
at com.google.devtools.build.lib.remote.RemoteSpawnRunner.exec(RemoteSpawnRunner.java:266)
at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:159)
at com.google.devtools.build.lib.exec.AbstractSpawnStrategy.exec(AbstractSpawnStrategy.java:119)
at com.google.devtools.build.lib.exec.SpawnStrategyResolver.exec(SpawnStrategyResolver.java:45)
at com.google.devtools.build.lib.rules.cpp.CppCompileAction.execute(CppCompileAction.java:1402)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.executeAction(SkyframeActionExecutor.java:1144)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor$ActionRunner.run(SkyframeActionExecutor.java:1061)
at com.google.devtools.build.lib.skyframe.ActionExecutionState.runStateMachine(ActionExecutionState.java:165)
at com.google.devtools.build.lib.skyframe.ActionExecutionState.getResultOrDependOnFuture(ActionExecutionState.java:94)
at com.google.devtools.build.lib.skyframe.SkyframeActionExecutor.executeAction(SkyframeActionExecutor.java:558)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.checkCacheAndExecuteIfNeeded(ActionExecutionFunction.java:859)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.computeInternal(ActionExecutionFunction.java:333)
at com.google.devtools.build.lib.skyframe.ActionExecutionFunction.compute(ActionExecutionFunction.java:171)
at com.google.devtools.build.skyframe.AbstractParallelEvaluator$Evaluate.run(AbstractParallelEvaluator.java:461)
at com.google.devtools.build.lib.concurrent.AbstractQueueVisitor$WrappedRunnable.run(AbstractQueueVisitor.java:414)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
at io.grpc.Status.asRuntimeException(Status.java:535)
at io.grpc.stub.ClientCalls$BlockingResponseStream.hasNext(ClientCalls.java:660)
at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$2(GrpcRemoteExecutor.java:175)
at com.google.devtools.build.lib.remote.Retrier.execute(Retrier.java:245)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:127)
at com.google.devtools.build.lib.remote.RemoteRetrier.execute(RemoteRetrier.java:116)
at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.lambda$executeRemotely$3(GrpcRemoteExecutor.java:146)
at com.google.devtools.build.lib.remote.util.Utils.refreshIfUnauthenticated(Utils.java:528)
at com.google.devtools.build.lib.remote.GrpcRemoteExecutor.executeRemotely(GrpcRemoteExecutor.java:144)
... 26 more
With these flags it seems to be more consistently reproducible:
CACHE=$(kubectl get gtw cache -o=jsonpath='{.status.addresses[0].value}')
SCHEDULER=$(kubectl get gtw scheduler -o=jsonpath='{.status.addresses[0].value}')
if [[
"$1" == "build" ||
"$1" == "coverage" ||
"$1" == "run" ||
"$1" == "test"
]]; then
bazelisk $1 \
--config=nativelink \
--grpc_keepalive_time=10s \
--experimental_remote_execution_keepalive=true \
--remote_upload_local_results=false \
--remote_cache=grpc://$CACHE:50051 \
--remote_executor=grpc://$SCHEDULER:50052 \
''${@:2}
else
bazelisk $@
fi
Fails after ~1min with:
INFO: Invocation ID: 03c25e4d-820d-4da2-9ddd-821673c9aa15
INFO: Analyzed 63 targets (0 packages loaded, 0 targets configured).
ERROR: /home/aaron/.cache/bazel/_bazel_aaron/7532c8de286b66ea1aaee4ff51952e7c/external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/BUILD.bazel:183:11: LlPr
ecompileModuleInterfaceUnit external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/modules/std/libcxx/vector.pcm failed: Exec failed due to IOException: 5 er
rors during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
ERROR: /home/aaron/.cache/bazel/_bazel_aaron/7532c8de286b66ea1aaee4ff51952e7c/external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/BUILD.bazel:183:11: LlPr
ecompileModuleInterfaceUnit external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/modules/std/libcxx/version.pcm failed: Exec failed due to IOException: 5 e
rrors during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
ERROR: /home/aaron/.cache/bazel/_bazel_aaron/7532c8de286b66ea1aaee4ff51952e7c/external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/BUILD.bazel:183:11: LlPr
ecompileModuleInterfaceUnit external/llvm-project-overlay~~llvm_project_overlay~llvm-project/libcxx/modules/std/libcxx/new.pcm failed: Exec failed due to IOException: 5 error
s during bulk transfer:
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
java.io.IOException: io.grpc.StatusRuntimeException: UNAVAILABLE: RST_STREAM closed stream. HTTP/2 error code: NO_ERROR
Note that the implementations of LlPrecompileModuleInterfaceUnit look explicitly disable remote execution due to known hermeticity issues:
ctx.actions.run(
outputs = [file_out, cdf_out],
inputs = compile_object_inputs(
ctx,
in_file,
headers,
bmis,
),
executable = toolchain.cpp_driver,
tools = compile_object_tools(ctx),
arguments = compile_object_args(
ctx,
in_file,
file_out,
cdf_out,
defines,
includes,
angled_includes,
bmis,
),
mnemonic = "LlPrecompileModuleInterfaceUnit",
execution_requirements = {
# Required so that module paths do not depend on the specific
# sandbox instance used during precompilation.
"no-sandbox": "1",
"no-remote": "1",
},
use_default_shell_env = False,
env = compile_object_environment(ctx),
)
This consistently failing at an action like this seems to suggest that either some Bazel flag is missing, or we need some server-side configuration to handle the case when an action is run locally. I'm also somewhat confused that these actions seems to be attempted remotely in the first place?
Continually invoking the bazel script above raises these kinds of errors at some point:
ERROR: /home/aaron/.cache/bazel/_bazel_aaron/7532c8de286b66ea1aaee4ff51952e7c/external/rules_ll~~rules_ll_dependencies~rocclr/BUILD.bazel:6:11: LlCompileObject external/rules
_ll~~rules_ll_dependencies~rocclr/thread/rocclr/monitor.o failed: (Exit 34): 419 errors during bulk transfer:
java.io.IOException: Error while uploading artifact with digest 'ccf7dace95c65984ed47ac5393c3c0542cc375cc830825cd370214a06239e120/23478'
java.io.IOException: Error while uploading artifact with digest 'c0562fe4469fcc9162d1dcc7c920fa861d2b44d55747ea94b44aed23e25335f1/30588'
java.io.IOException: Error while uploading artifact with digest '90c491f320fe7d5d3cd47dea692b6b5681b9b9110616e22e3db5a7e5a694dd6a/3523'
java.io.IOException: Error while uploading artifact with digest 'ec5ad9d9c4f0ac272d15739de31b144fa075bc9d85cb9e288140faa489d0bdaa/4302696'
java.io.IOException: Error while uploading artifact with digest '2cdff180f15d821dc3373fe821456e2ff00c6f80f7c48c6ed8796711582923a3/2291'
j
,,, # hundreds more of these messages...
at com.google.devtools.build.lib.remote.util.RxUtils$BulkTransferExceptionCollector.onResult(RxUtils.java:112)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableCollectSingle$CollectSubscriber.onNext(FlowableCollectSingle.java:94)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onNext(SingleFlatMapPublisher.java:107)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onNext(FlowableUsing.java:104)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapPublisher$SingleFlatMapPublisherObserver.onNext(SingleFlatMapPublisher.java:107)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableUsing$UsingSubscriber.onNext(FlowableUsing.java:104)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber.innerSuccess(FlowableFlatMapSingle.java:173)
at io.reactivex.rxjava3.internal.operators.flowable.FlowableFlatMapSingle$FlatMapSingleSubscriber$InnerObserver.onSuccess(FlowableFlatMapSingle.java:342)
at io.reactivex.rxjava3.internal.observers.ResumeSingleObserver.onSuccess(ResumeSingleObserver.java:46)
at io.reactivex.rxjava3.internal.operators.single.SingleJust.subscribeActual(SingleJust.java:30)
at io.reactivex.rxjava3.core.Single.subscribe(Single.java:4855)
at io.reactivex.rxjava3.internal.operators.single.SingleResumeNext$ResumeMainSingleObserver.onError(SingleResumeNext.java:80)
at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
at io.reactivex.rxjava3.internal.operators.completable.CompletableFromObservable$CompletableFromObservableObserver.onError(CompletableFromObservable.java:51)
at io.reactivex.rxjava3.subjects.AsyncSubject$AsyncDisposable.onError(AsyncSubject.java:354)
at io.reactivex.rxjava3.subjects.AsyncSubject.onError(AsyncSubject.java:171)
at com.google.devtools.build.lib.remote.RemoteExecutionCache$1.onError(RemoteExecutionCache.java:234)
at io.reactivex.rxjava3.internal.operators.completable.CompletableFromSingle$CompletableFromSingleObserver.onError(CompletableFromSingle.java:41)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
at com.google.devtools.build.lib.remote.util.AsyncTaskCache$1.onError(AsyncTaskCache.java:339)
at com.google.devtools.build.lib.remote.util.AsyncTaskCache$Execution.onError(AsyncTaskCache.java:205)
at io.reactivex.rxjava3.internal.operators.completable.CompletableToSingle$ToSingle.onError(CompletableToSingle.java:73)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMapCompletable$FlatMapCompletableObserver.onError(SingleFlatMapCompletable.java:97)
at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.tryOnError(CompletableCreate.java:91)
at io.reactivex.rxjava3.internal.operators.completable.CompletableCreate$Emitter.onError(CompletableCreate.java:77)
at com.google.devtools.build.lib.remote.util.RxFutures$OnceCompletableOnSubscribe$1.onFailure(RxFutures.java:102)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1119)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:125)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setFuture(AbstractFuture.java:850)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:216)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.setResult(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:144)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.common.util.concurrent.AbstractTransformFuture.run(AbstractTransformFuture.java:105)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at com.google.common.util.concurrent.SettableFuture.setException(SettableFuture.java:55)
at com.google.devtools.build.lib.remote.util.RxFutures$2.onError(RxFutures.java:259)
at io.reactivex.rxjava3.internal.operators.single.SingleFlatMap$SingleFlatMapCallback$FlatMapSingleObserver.onError(SingleFlatMap.java:117)
at io.reactivex.rxjava3.internal.operators.single.SingleUsing$UsingSingleObserver.onError(SingleUsing.java:180)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.tryOnError(SingleCreate.java:95)
at io.reactivex.rxjava3.internal.operators.single.SingleCreate$Emitter.onError(SingleCreate.java:81)
at com.google.devtools.build.lib.remote.util.RxFutures$OnceSingleOnSubscribe$1.onFailure(RxFutures.java:172)
at com.google.common.util.concurrent.Futures$CallbackListener.run(Futures.java:1127)
at com.google.common.util.concurrent.DirectExecutor.execute(DirectExecutor.java:31)
at com.google.common.util.concurrent.AbstractFuture.executeListener(AbstractFuture.java:1286)
at com.google.common.util.concurrent.AbstractFuture.complete(AbstractFuture.java:1055)
at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:807)
at io.grpc.stub.ClientCalls$GrpcFuture.setException(ClientCalls.java:572)
at io.grpc.stub.ClientCalls$UnaryStreamToFuture.onClose(ClientCalls.java:542)
at io.grpc.internal.ClientCallImpl.closeObserver(ClientCallImpl.java:562)
at io.grpc.internal.ClientCallImpl.access$300(ClientCallImpl.java:70)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInternal(ClientCallImpl.java:743)
at io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1StreamClosed.runInContext(ClientCallImpl.java:722)
at io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:133)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
Suppressed: java.io.IOException: Error while uploading artifact with digest 'ccf7dace95c65984ed47ac5393c3c0542cc375cc830825cd370214a06239e120/23478'
at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$0(ByteStreamUploader.java:176)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:205)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
... 73 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: not found : Failed on query_write_status() command
at io.grpc.Status.asRuntimeException(Status.java:535)
... 10 more
Suppressed: java.io.IOException: Error while uploading artifact with digest 'c0562fe4469fcc9162d1dcc7c920fa861d2b44d55747ea94b44aed23e25335f1/30588'
at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$0(ByteStreamUploader.java:176)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:205)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
... 73 more
Caused by: io.grpc.StatusRuntimeException: NOT_FOUND: not found : Failed on query_write_status() command
at io.grpc.Status.asRuntimeException(Status.java:535)
... 10 more
Suppressed: java.io.IOException: Error while uploading artifact with digest '90c491f320fe7d5d3cd47dea692b6b5681b9b9110616e22e3db5a7e5a694dd6a/3523'
at com.google.devtools.build.lib.remote.ByteStreamUploader.lambda$uploadBlobAsync$0(ByteStreamUploader.java:176)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:205)
at com.google.common.util.concurrent.AbstractCatchingFuture$AsyncCatchingFuture.doFallback(AbstractCatchingFuture.java:192)
at com.google.common.util.concurrent.AbstractCatchingFuture.run(AbstractCatchingFuture.java:134)
... 73 more
... # hundreds more of these messages...