bazel-buildfarm
bazel-buildfarm copied to clipboard
Published
20 hours ago •
bazelbuild
bazelbuild
Bazel hangs indefinitely when Buildfarm workers go away
Observed the following in our deployment of 1.11.2:
Build is in flight, during the build, 1 or more workers are scaled in due to low cluster utilization. Bazel hangs executing a target indefinitely until build is cancelled.
Below is the log in asc time from one of the schedulers. It appears that when we fail to fetch one or more inputs the operation hangs and does not send a proper response back to bazel, which appears to be waiting for an execution.
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4bd9cb25-1958-4f01-8e76-c9d1ae596213) 10.35.216.154:8981: 15 remaining of 16 516us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4bd9cb25-1958-4f01-8e76-c9d1ae596213) 10.35.213.104:8981: 15 remaining of 16 59999597us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 16
[WARNING] build.buildfarm.instance.server.AbstractServerInstance$3 onFailure - expect for 09db6fc75d87bf46335d9b2ea9f6fb3cb4db4251/3992 failed
[WARNING] com.github.benmanes.caffeine.cache.LocalAsyncCache lambda$handleCompletion$7 - Exception thrown during asynchronous load
[WARNING] build.buildfarm.instance.shard.ShardInstance$14 onFailure - an rpc status was thrown with DEADLINE_EXCEEDED for shard/operations/cd4e29c1-c713-403d-8cba-e0c1f4eeb085, discarding it
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$1$1 onFailure - error queueing shard/operations/cd4e29c1-c713-403d-8cba-e0c1f4eeb085
[WARNING] build.buildfarm.instance.server.AbstractServerInstance$3 onFailure - expect for 097544fe4b82d5299b0ee67ef99bfb5f0f273efb/139 failed
[WARNING] com.github.benmanes.caffeine.cache.LocalAsyncCache lambda$handleCompletion$7 - Exception thrown during asynchronous load
[SEVERE ] com.google.common.util.concurrent.AggregateFuture log - An additional input failed after the first. Logging it after adding the first failure as a suppressed exception.
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 736c6b51c207149c3b8b255d1f768faccd78c0ce/944 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 736c6b51c207149c3b8b255d1f768faccd78c0ce/944(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 30079e2259792f637fcad7be9574a83c5eee1bc8/15433088 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 30079e2259792f637fcad7be9574a83c5eee1bc8/15433088(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for cd8b2dabe6f5cbddb17bb7efed15dac8a24f21e6/777 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading cd8b2dabe6f5cbddb17bb7efed15dac8a24f21e6/777(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.210.113:8981: 2466 remaining of 19023 24323us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.217.201:8981: 2171 remaining of 19023 3502us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.211.126:8981: 2034 remaining of 19023 3255us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.216.58:8981: 1904 remaining of 19023 3336us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.216.154:8981: 1781 remaining of 19023 3209us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.219.19:8981: 1597 remaining of 19023 2869us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.219.117:8981: 1501 remaining of 19023 2846us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.220.78:8981: 1306 remaining of 19023 2629us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(1057213c-aeac-474c-a09e-b7ffd29e5415) 10.35.213.104:8981: 1306 remaining of 19023 59932388us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 19024
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 8c5a11fade22d3c828e56aafffbf2f1ec19c76b2/682 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 8c5a11fade22d3c828e56aafffbf2f1ec19c76b2/682(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 6c8cdc193159089a95c397d2287f53792e7dfa3a/472 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 6c8cdc193159089a95c397d2287f53792e7dfa3a/472(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 10e717e6a03513d1db1333fb101414487b6a768d/783(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 81e9977190d9f428619441e6879b165e6408ad55/1576 on 10.35.213.104:8981
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 81e9977190d9f428619441e6879b165e6408ad55/1576(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 0e7a544d4b528715b526bde86e621444d2c5f498/749 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 0e7a544d4b528715b526bde86e621444d2c5f498/749(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 466e5ab792152733eea110435072bc477dd54da8/571 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 466e5ab792152733eea110435072bc477dd54da8/571(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 6ad01e89427c55d80aef5bfbf47564a69e685d1a/766 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 6ad01e89427c55d80aef5bfbf47564a69e685d1a/766(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(72ae7c77-40df-4f5b-b5bb-118d466c4dea) 10.35.220.135:8981: 16 remaining of 17 699us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(72ae7c77-40df-4f5b-b5bb-118d466c4dea) 10.35.223.189:8981: 15 remaining of 17 567us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(72ae7c77-40df-4f5b-b5bb-118d466c4dea) 10.35.213.104:8981: 15 remaining of 17 59999726us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 17
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for c26d2b6282a20552f5bff6d79446513ea8b73624/788 on 10.35.213.104:8981
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading c26d2b6282a20552f5bff6d79446513ea8b73624/788(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.216.195:8981: 18 remaining of 19 642us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.219.19:8981: 17 remaining of 19 688us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.217.10:8981: 17 remaining of 19 601us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.208.122:8981: 17 remaining of 19 457us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.217.201:8981: 17 remaining of 19 706us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.216.58:8981: 15 remaining of 19 739us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.220.151:8981: 14 remaining of 19 676us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.219.117:8981: 14 remaining of 19 636us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.216.154:8981: 14 remaining of 19 496us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.223.189:8981: 14 remaining of 19 567us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.210.113:8981: 14 remaining of 19 527us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.220.78:8981: 14 remaining of 19 579us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.213.200:8981: 12 remaining of 19 603us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.212.131:8981: 12 remaining of 19 622us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(6cd6c0c5-31bb-4f8b-a841-27a17f3e2025) 10.35.213.104:8981: 12 remaining of 19 59990972us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 19
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.217.201:8981: 15 remaining of 16 674us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.223.189:8981: 15 remaining of 16 688us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.216.58:8981: 13 remaining of 16 654us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.215.62:8981: 13 remaining of 16 771us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.219.117:8981: 13 remaining of 16 500us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.208.122:8981: 13 remaining of 16 497us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.216.195:8981: 13 remaining of 16 590us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(e789196e-2e6d-4ec4-b641-8abd3c762c33) 10.35.213.104:8981: 13 remaining of 16 59995418us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 16
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.210.113:8981: 16 remaining of 17 666us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.217.10:8981: 16 remaining of 17 735us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.216.58:8981: 14 remaining of 17 627us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.216.195:8981: 14 remaining of 17 654us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.216.154:8981: 14 remaining of 17 632us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.213.200:8981: 13 remaining of 17 598us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5bf27e32-fbdf-42c1-95a2-79e2f8a8d7fe) 10.35.213.104:8981: 13 remaining of 17 59995862us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 17
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 7d6b618bd7161b7854391835718ebee96aa76153/783 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7d6b618bd7161b7854391835718ebee96aa76153/783(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.220.135:8981: 16 remaining of 17 709us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.213.200:8981: 15 remaining of 17 697us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.211.8:8981: 15 remaining of 17 597us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.217.201:8981: 15 remaining of 17 535us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.210.113:8981: 15 remaining of 17 581us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.212.131:8981: 15 remaining of 17 547us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.216.154:8981: 15 remaining of 17 647us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.215.62:8981: 15 remaining of 17 714us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.211.126:8981: 15 remaining of 17 552us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.220.78:8981: 14 remaining of 17 733us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.223.58:8981: 14 remaining of 17 569us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.208.122:8981: 14 remaining of 17 565us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.219.117:8981: 14 remaining of 17 661us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.219.19:8981: 14 remaining of 17 738us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(75c489c1-ff43-479f-967f-ae3bf99f8835) 10.35.213.104:8981: 14 remaining of 17 59990849us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 17
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.213.200:8981: 14 remaining of 16 729us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.223.58:8981: 14 remaining of 16 644us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.211.126:8981: 14 remaining of 16 637us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.223.189:8981: 14 remaining of 16 762us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.216.58:8981: 12 remaining of 16 639us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.219.117:8981: 12 remaining of 16 614us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(955d2102-4879-45eb-b9ee-7427c89b8822) 10.35.213.104:8981: 12 remaining of 16 59995789us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 16
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for a41ec6966c66242aff6cb874ea29a42e84a37a7d/242464 on 10.35.213.104:8981
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading a41ec6966c66242aff6cb874ea29a42e84a37a7d/242464(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for b693e552818e7efcaf0951723a426ee831e1abda/753 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b693e552818e7efcaf0951723a426ee831e1abda/753(shard) at offset 0
[WARNING] build.buildfarm.server.WriteStreamObserver errorResponse - bazel-4.2.2rc1: f346f130-8c49-43de-a33b-c6a1e11b85b2 -> a9cda198-673d-4d90-a0dd-c66c93748936 -> 5009c1599c776dbf34a59fbc1ef79e6f667b937a: exceeded entry limit for uploads/50ba63f6-09db-45e3-a0fc-41cf23fe09b8/blobs/89345610197589455fab55acdcee861d5247eb4f/3701780480
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 72c0f27e8b02bc132bfd939e364cb4e745fd400a/709 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 72c0f27e8b02bc132bfd939e364cb4e745fd400a/709(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 3719cd4910f6cd089faf73c1a2ab1ba807830be1/556(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 32f549cc69f603baa3c05801bf3881188e647a51/1069(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading c0454081ae330a3bb288b6a312769d562181fc8d/4326(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 839263989d21bcc9c25aa087364c0a3f442cc78b/14712(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 6f7bdc59eedd6a2c9c3b70929a79233094224318/396(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 667bdc558ff8c62001d38a109921f0f8c5963bbe/329(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 87a95e056011349e81fa3a46e53fa556ae99e6fa/9580(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 618704ef6393f99e67236117c4216eb3cd627ccd/621(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 67afcdbfc54b3e70753f713751afd9a2e74390f3/318(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f0cf8c4469310c641ba564e99d1070703130682a/1476(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7086bfc60c684b25bcaf4340a6cbb0f142e035f6/20414(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading d6ec2cb9bfac486ed8ab7f6e745911e6c085ef3d/4933(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 46b1cf45b760a4c36fa5cedc5d8bc1afd20f8ad7/3778(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading e4a25aeddee983c758d6f7ede27d7642f51db983/3341(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 85d8d8604d02b7b7ef8eb73a8d488dee2425d598/36837(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 4417c39c18d80547bb081a9f4f3727afb202291d/2191(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 0cc6b075d209c32fbd2a06c218fff0d0cc70078f/6678(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading e81ad2fcc6ec3a3eaa08163dd847650cf3015490/1332(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7caff836ed0635e143fc0f7836b4a2773cb06fe8/2350(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 64229501f6c066c2a9cb9c547bfe31ad74da62d1/9254(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 548b65f6211bf37f4f1bc368f33a755aeb7af592/15525(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 5ff9496a7e51fb43d32514986055186f4a0558aa/17832(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 70876b487d1e31261a68ee2d4c21920d9f67ef35/16728(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading c10f0c5b2eec69ce864454b50bef6a06255124f1/15885(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 9c6e96cba6d10b6adf92355a38748163ade8cb0b/168712(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 969bec067365bc6b1ca1acd698039ef6b022486e/11307 on 10.35.213.104:8981
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 969bec067365bc6b1ca1acd698039ef6b022486e/11307(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 8cf006e05d773bbcc6016be72dcfc23ae7856fa6/26579 on 10.35.213.104:8981
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 8cf006e05d773bbcc6016be72dcfc23ae7856fa6/26579(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading d14eef2c15794e8e3f282203b2dec0fbdb630edc/7911(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 680d795220ecf670ea3a5eebe241292ac7f22669/8192(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 1819c4ec70d80f15a78bb3d5c63b76a2121af9e2/1459(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 34d33cd12de734327857f0769b29e86da51ec45c/54912(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 4b4222001fb385fcc30cbe71d99004b98ee04c87/4872(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f32dab3880d47eca1b71c308cf6542b32941b23c/7127(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading db9e88b8332953972c9120c73389fa2ce03dd8f8/30562(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 5d8a9c42f902d2418a7e63b539c0f319373e9400/8472(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading cd58a593934f31e39568f0c5c05ad4ee3734f475/6129(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 6ce2245576c03d53572c9b4ae496c33d7f34cca2/1592(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 3568fc32cc3d07f0a2a377f7fef75105bc1f4fb2/9763(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading befedf222d752d445aa7870d8df499ca86457742/10556(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 1200041690ae3025636e0f56bf0b7c53a13eb9b4/10147(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 1ec3378d6dcad5ea86a85e7ba5cee7fda97a7284/12053(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading f6106880a8669bf4a2c24abee195e6debbb6f5a8/17429(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading c6d110114d1b9cfa70f80abaf5757bb17f999321/47610(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 41133430938a3455e36b16fe385217a39345c965/256272(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading ae2c2a282ddc7159dae89126371cb6c4aea33696/15643(shard) at offset 0
[WARNING] build.buildfarm.server.WriteStreamObserver errorResponse - bazel-4.2.2rc1: f346f130-8c49-43de-a33b-c6a1e11b85b2 -> a9cda198-673d-4d90-a0dd-c66c93748936 -> 1a5fe06ef3db960c565e26c50da1ee7ef8c60ec1: exceeded entry limit for uploads/c8dc377c-d720-4ef0-aecc-6f6faf70ea91/blobs/3c1ab4fe15d5039484f85caa172fdc70c768a893/5096683520
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 16685168672a398469f9590db79a4484ecdc2660/994(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 948cfaca3e3cc7f5da7784592dc2202f4e856d25/10905(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading eaa4569dcf456e7f12ee881d9b0f5842a9c9c91d/25808(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 2a33310d9c1e394e66b00f1bbc34a105f11c14c8/14288(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading c881671a6c893f57610e5191747153e87cc9db1c/612840(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7ed3bc55f25a4c7c17543a3a2a1f0689b61183ca/1378336(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading d5d65b5e642a1b501b58254510c66d134bbdd78a/225312(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 0ac772f8b8e0c0be3adf14373419b425bc2631fd/519024(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 8e8268b0ec8451582b2a487fdff2ffc2af6897a4/337(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 84d60f178824d1f3f37fe389bf1f4a2a9ba4ea12/2808(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 7e2d52c27debcecebbebdc0ad21c481309f99105/504192(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 0645c9e957cf4c1ab7f44fc921ef6446eb7a65e7/162672(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading a4b1689f2eecb74cf1940c1094e74268aa0c62da/232832(shard) at offset 0
[WARNING] build.buildfarm.server.WriteStreamObserver errorResponse - bazel-4.2.2rc1: 34b5adab-b6c7-4e39-986e-8fcbb6849697 -> 4b921e49-71d6-46da-9dd3-e8e110cdbaeb -> 612f93f9db6911bd5f4ebff142a7f187b39b6bb9: exceeded entry limit for uploads/b5bf0e2f-2d3a-4a3f-a2d0-63c3da5b53fa/blobs/18e88fa604f0c90968fbe91b5ef76f018eb910a9/4467937280
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading e23246ad12cd510855fc13b7009cc079c3d5500d/37157(shard) at offset 0
[WARNING] build.buildfarm.server.WriteStreamObserver errorResponse - bazel-4.2.2rc1: 34b5adab-b6c7-4e39-986e-8fcbb6849697 -> 4b921e49-71d6-46da-9dd3-e8e110cdbaeb -> 710e817f6d4d8abd317993c43aacbe74f7e3e61b: exceeded entry limit for uploads/00180a7e-62c2-4807-bdb9-86d9511b8625/blobs/fddd5f6b2da05a66041db08c65bf7f78f287574b/5076387840
[WARNING] build.buildfarm.server.WriteStreamObserver errorResponse - bazel-4.2.2rc1: 34b5adab-b6c7-4e39-986e-8fcbb6849697 -> 4b921e49-71d6-46da-9dd3-e8e110cdbaeb -> 144614211f77ff9a57d2959214d5759806bde504: exceeded entry limit for uploads/c3d58893-5ee2-40ba-a750-da11a724c60b/blobs/fddd5f6b2da05a66041db08c65bf7f78f287574b/5076387840
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 083e72121e75cc0cec129283c99de1ca95bb4f9e/20956(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 4b0537c5a498dce48045172c64154930141805c5/79809(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading bca516cefb2c202b7b8f6d3ac855a60af4cf0cd5/43937(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b54b49babe4e79817b7c54f41f2ea89b968b6b73/15318(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading a5e3a54ba269e95ee91d9f5c69fdf3e1ddc2f6fb/70536(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f4255b1fa938e79a0820fb4a4753b58c33a3cae3/18276(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 72f146f1df130d2c89de6d2ade1cb01f7aeca095/25521(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 62fe7ef46df29977bef00ca324735755a8556a47/399893(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 0fb45a1de965cae4b9722344d4b00b9b903d4f36/21761(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 4b4880e3dd51c147ac9596c6fec52f94a66be4d1/15155(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b7c5d1a7ff3226e9f2e99a88bcd0804bb38f7744/4745(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f6242c76a381272240ca31f8475955b6cbb45b3d/14368(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading d28139905820afca5f8a84b5cb10a3df16193716/15033(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f9d8331303f4c86e6ce9eed3ae9eec0da6331fc9/204356(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0b5739d5-1239-422e-98bd-1da4c696d1ce) 10.35.216.195:8981: 16 remaining of 20 837us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0b5739d5-1239-422e-98bd-1da4c696d1ce) 10.35.213.104:8981: 16 remaining of 20 59999175us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 20
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f88b3bea6586d19cb00b5395167d1f7f2bc9af47/57608(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 2fd0a86b7b33cd92f71730f7517a3e04f8f7ddc4/5898152 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 2fd0a86b7b33cd92f71730f7517a3e04f8f7ddc4/5898152(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bdc94b20-e236-49ab-b28e-5794fed2718c) 10.35.213.104:8981: 19 remaining of 19 60000725us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 19
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 430146b076f1132260aefd5ca77f474ca95a5254/19098(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7f9a5627338c08555933f3c6932ca37ce8cb5edc/43889(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.219.117:8981: 18 remaining of 22 848us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.216.154:8981: 17 remaining of 22 540us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.213.200:8981: 16 remaining of 22 587us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.217.10:8981: 15 remaining of 22 632us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.210.113:8981: 15 remaining of 22 703us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.220.151:8981: 13 remaining of 22 619us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.223.58:8981: 11 remaining of 22 592us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.216.58:8981: 10 remaining of 22 580us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.223.189:8981: 9 remaining of 22 794us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.211.8:8981: 8 remaining of 22 703us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.219.19:8981: 6 remaining of 22 822us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.208.122:8981: 6 remaining of 22 833us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(bfd6cde3-6c86-4a88-96da-e8e4fa8d7f12) 10.35.213.104:8981: 6 remaining of 22 59991846us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 22
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.223.189:8981: 15 remaining of 19 820us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.219.117:8981: 15 remaining of 19 1048us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.219.19:8981: 14 remaining of 19 842us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.217.10:8981: 13 remaining of 19 888us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.215.62:8981: 13 remaining of 19 816us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.212.131:8981: 13 remaining of 19 801us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.216.58:8981: 12 remaining of 19 314us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(5449363d-d228-4f2c-8286-0b31535e8cf8) 10.35.213.104:8981: 12 remaining of 19 59994805us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 19
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading d131e49b74552c1d7206eec9a34f687bb7d638cb/13914(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7bb90ca8bfe13428f958b8f52127cb5336185037/4658(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b1d3471654bf4f148b126e188979a7fa4c99686d/17389(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 7433ad367ba35c12df08ebcebe114b41625fcb0d/26809(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.216.195:8981: 17 remaining of 25 615us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.212.131:8981: 17 remaining of 25 835us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.215.62:8981: 16 remaining of 25 1148us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.208.122:8981: 16 remaining of 25 846us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.211.126:8981: 16 remaining of 25 668us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.217.201:8981: 15 remaining of 25 431us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.211.8:8981: 13 remaining of 25 597us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.220.78:8981: 12 remaining of 25 879us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.223.58:8981: 11 remaining of 25 797us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.216.58:8981: 9 remaining of 25 532us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.219.117:8981: 9 remaining of 25 808us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.219.19:8981: 8 remaining of 25 849us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.220.151:8981: 7 remaining of 25 693us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(d82e44fc-9c7e-43f5-bc3f-402e5bb83966) 10.35.213.104:8981: 7 remaining of 25 59990074us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 25
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 5e72afaa9a72f748d2d279ac2ce5b1c86ecc2518/21482(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 0d12bdb2e0425380d80cf73a3fc513e1eaca4b39/19589(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.208.122:8981: 18 remaining of 21 870us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.211.8:8981: 17 remaining of 21 664us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.219.19:8981: 16 remaining of 21 576us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.211.126:8981: 16 remaining of 21 576us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.210.113:8981: 16 remaining of 21 608us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.212.131:8981: 16 remaining of 21 859us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.219.117:8981: 16 remaining of 21 691us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.217.10:8981: 15 remaining of 21 940us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.215.62:8981: 15 remaining of 21 951us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.220.135:8981: 13 remaining of 21 763us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.223.189:8981: 12 remaining of 21 791us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.217.201:8981: 12 remaining of 21 900us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.223.58:8981: 11 remaining of 21 704us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.216.154:8981: 10 remaining of 21 628us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.216.195:8981: 8 remaining of 21 804us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(f7c34d47-8b39-4be6-9314-8294ba5791a1) 10.35.213.104:8981: 8 remaining of 21 59988285us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: ClientCall was cancelled at or after deadline. [remote_addr=/10.35.213.104:8981]
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 21
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.211.126:8981: 60 remaining of 77 904us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.210.113:8981: 56 remaining of 77 792us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.212.131:8981: 51 remaining of 77 838us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.219.19:8981: 48 remaining of 77 615us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.219.117:8981: 48 remaining of 77 816us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.220.78:8981: 40 remaining of 77 895us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.215.62:8981: 37 remaining of 77 922us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.216.154:8981: 31 remaining of 77 673us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.220.151:8981: 28 remaining of 77 889us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.213.200:8981: 27 remaining of 77 495us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.217.10:8981: 25 remaining of 77 819us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.220.135:8981: 25 remaining of 77 607us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.223.58:8981: 23 remaining of 77 528us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.208.122:8981: 18 remaining of 77 853us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.216.195:8981: 15 remaining of 77 638us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(4c39ab12-475a-4c6c-8894-c528cbc5da1f) 10.35.213.104:8981: 15 remaining of 77 59988711us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 77
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 0d1dfcb21dc9eca0f2ce42a44be949cca31404f3/16860(shard) at offset 0
[SEVERE ] build.buildfarm.instance.shard.Util$3 onFailure - error checking for 561ff130ec63a96ae4866e46bccb96159bc01887/5872856 on 10.35.213.104:8981
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 561ff130ec63a96ae4866e46bccb96159bc01887/5872856(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 6a698849dc77deba6c195445a3ecd91d22d0025a/139106(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 522f11f49122fe37fc7094581ec07057745047e5/71977(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 6f2391f79021afef7d28188a125676a8f3822464/318300(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b105db4555a970735599a6d550afcc9c5fca64da/34820(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading aa5246a88e4824e7472c288d7521c150383e707e/28648(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 9e481c1dcc962c3c0bf1bb636606d0218202d445/26010(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading eb63981b0162e395e2f5da4889dd196c45e31499/45123(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.216.58:8981: 17 remaining of 20 600us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.220.78:8981: 15 remaining of 20 828us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.223.189:8981: 14 remaining of 20 837us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.212.131:8981: 14 remaining of 20 671us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.210.113:8981: 14 remaining of 20 716us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.216.195:8981: 13 remaining of 20 932us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.213.200:8981: 12 remaining of 20 773us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.215.62:8981: 12 remaining of 20 842us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.220.135:8981: 12 remaining of 20 572us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.219.117:8981: 12 remaining of 20 735us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.217.201:8981: 9 remaining of 20 725us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.217.10:8981: 7 remaining of 20 664us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.216.154:8981: 5 remaining of 20 675us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.223.58:8981: 4 remaining of 20 712us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.211.8:8981: 3 remaining of 20 545us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(0a87a990-b23e-447a-bb78-3a7e9a7df455) 10.35.213.104:8981: 3 remaining of 20 59988822us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 20
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 77391b478755b9583f315f4cc49d73cae5ee267a/35078(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.219.19:8981: 16 remaining of 20 884us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.217.201:8981: 16 remaining of 20 764us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.216.195:8981: 15 remaining of 20 665us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.223.189:8981: 14 remaining of 20 931us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.220.151:8981: 12 remaining of 20 974us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.216.154:8981: 11 remaining of 20 859us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.213.200:8981: 10 remaining of 20 593us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.220.78:8981: 8 remaining of 20 735us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.208.122:8981: 7 remaining of 20 737us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.216.58:8981: 6 remaining of 20 622us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.210.113:8981: 6 remaining of 20 400us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.217.10:8981: 4 remaining of 20 702us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.211.8:8981: 3 remaining of 20 634us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.211.126:8981: 3 remaining of 20 827us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.223.58:8981: 2 remaining of 20 430us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.220.135:8981: 2 remaining of 20 633us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.215.62:8981: 2 remaining of 20 741us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(de9e2033-cb6e-441c-971c-014f290f88ca) 10.35.213.104:8981: 2 remaining of 20 59987361us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 20
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading c54ab3eb845e33f3d62b338347803f406630ab90/28881(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading ad7e6598b7bca6d13d3e108743e961b8a0312d0f/29034(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 236eb838d895c2fea5a61ac44ee8e2ea97e095b5/35687(shard) at offset 0
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.217.10:8981: 14 remaining of 20 379us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.220.151:8981: 14 remaining of 20 740us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.219.117:8981: 14 remaining of 20 612us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.210.113:8981: 14 remaining of 20 664us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.211.8:8981: 14 remaining of 20 677us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.223.189:8981: 13 remaining of 20 827us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.216.58:8981: 13 remaining of 20 763us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.216.154:8981: 13 remaining of 20 823us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.217.201:8981: 13 remaining of 20 820us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.219.19:8981: 12 remaining of 20 756us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.211.126:8981: 11 remaining of 20 585us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.208.122:8981: 10 remaining of 20 931us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.215.62:8981: 9 remaining of 20 925us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.216.195:8981: 9 remaining of 20 734us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.213.200:8981: 7 remaining of 20 681us
[WARNING] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.223.58:8981: 6 remaining of 20 793us
[SEVERE ] build.buildfarm.instance.shard.ShardInstance$2 onFailure - DEADLINE_EXCEEDED: findMissingBlobs(feb3ac78-a384-4586-b256-99ce633b9484) 10.35.213.104:8981: 6 remaining of 20 59987914us: io.grpc.StatusRuntimeException: DEADLINE_EXCEEDED: context timed out
[SEVERE ] build.buildfarm.server.ContentAddressableStorageService$1 onFailure - findMissingBlobs(): 20
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 834084f782c6803bb5b1fde096034a1050180a79/38944(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 1773bf06ab84578076f270dd73ddacac05b8db3d/2933904(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 50bbbf36dfbdca84d861982dc1d80227709c3681/1457(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 3871e4c077703797cc26b3bd949ec71585fe6010/2834(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 3f4ee3dac926a7416b517d124abf367f01623e2b/4089(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 4c24f3f52097d6ae30b89aa9471d7dd96f659033/2321(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading f7f5a4d0a1743fdf9fc5dbe28efbaa17f762b207/2901(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 1cdf72ecfa3b476f77347236205ec5c0ade05109/9864(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 354d6b5ba70776be855391481c7e59eef467e6da/4269(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 4810db6a678013b61d1d5c98a112c1bfca8305fe/2612(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 15c935fec94f10f9a1d6970c841322ccc794828e/5853(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading b0de9b1bd1b11cc158ae3f34131b00f24855bcfb/1311(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 8e6e51ca180bc2eee154dbc8f16c116d589fc4ed/1758(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading c937e7251b123d5feed7abe1e12bda21445fc1c5/2344(shard) at offset 0
[WARNING] build.buildfarm.instance.server.AbstractServerInstance$3 onFailure - expect for b26a8f57f582129ae10dbc8a2cd1ff925b230ada/675831 failed
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading b5d85798be560980285973e6a4ce06bc4e18a7a6/5051(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading f1136f1d516f32a49c3f76bd5273c9a5f9f6172b/6638(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading f91bc107a6520074fe6443752489f2657d25ba81/8262(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading f7a0883de30a6916806ca0cb10eb9e29ae3ad152/1854(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 926b14b029c9b72812e03bbd393edce1710bebd6/959(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 014f49e85f017aa2c81f79c964e4ace95ebc5df4/682(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading d4ee665ecfc47aa93f35045c0e63e392de62b4f3/3287(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 41d4f3405d18cd35e5e4fbcacfa39fa59bbb4c25/926(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 883190482ef3057ab5445ef93a6d3969495eacb7/14663(shard) at offset 0
[WARNING] build.buildfarm.server.ByteStreamService$1 onError - error reading 8cb96d2a5d79b972f3d981227d27f2983a09193b/2724(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 2633e02ed4bd7c58757dd128237f9d7c7c8b1864/7059(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading a92ca347f90190e7b3d1ba32bf5d834b1d46aa89/31218(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading e2f70335b1d528097c20cce8472cff9d9982b6ef/33467(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading b63719466799d36b32c02d0311e38c19960f63db/17731(shard) at offset 0
[SEVERE ] build.buildfarm.server.ByteStreamService$1 onError - error reading 45d39e6b43b8d9ff99228d429f57a7f54fb5a179/51(shard) at offset 0
Bazel output:
(15:55:35) [32m[296,184 / 296,258][0m Compiling ...... [//........] [for .....]; 6055s remote
Hm, can I get you to bisect for 1.11.2 and kill off workers to try to find a commit?
Hm, can I get you to bisect for 1.11.2 and kill off workers to try to find a commit?
So I don't think it's a recent issue. I'm pretty sure this has existed for a long time. We've observed it previously with older version of bazel as actions were getting stuck in the Prepa state. Newer version of Bazel that fixed the display issue is now correctly showing these in the executing state.
I'm unable to replicate: killing off my workers results in completely transparent success from client builds.
I definitely get a problem when doing Ctrl-C to interrupt, where interrupts are being conveyed as the status of executing processes, and their executions are conveyed to the client, but nothing like what you've pasted here, and nothing that can't be overcome with a direct TERM/KILL on the java process.
Not able to get a stack trace yet but here is another piece of data:
1 worker scales down. At the same time, target //abc:xyz starts executing
//abc:xyz_test,CACHE_CHECK //abc:xyz_test,QUEUED //abc:xyz_test,EXECUTING
No COMPLETED piece of metadata ever received. Bazel build is stuck until it's timed out.