George Gensure

This is an extremely broad question. Buildfarm is an arbitrary code execution machine, so the surface area here is quite wide. Can you narrow down the scope of 'untrusted', and...

Even within 'execution' there are a number of vectors here. Cache poisoning is a risk via the frontend. #386 is meant to handle this by config at least as a...

I recommend taking a look at the --experimental_remote_grpc_log from bazel per our [troubleshooting](https://bazelbuild.github.io/bazel-buildfarm/docs/tools/troubleshooting-bazel-remote-execution/) docs. You should be able to look at the operation that failed and see whether there's anything...

Interesting. I never actually tried level 4, only theorized that it would work. 4 billion directories can break a lot of things. And no, this is a totally local setting...

> I saw 0-4 mentioned on: > [#568 (comment)](https://github.com/bazelbuild/bazel-buildfarm/issues/568#issuecomment-728223668) > > The same error can be reproduced with level 4 on MacOS workers too (level 2 works): > > ```...

Regardless we shouldn't be NPEing in any case. I'll look from my side

Nice find. This just makes me want to be definitive about my desire to have the shard cas be a real boy. I'll see about getting something up and in...

jstack for the stuck worker? Also, what's your buckets set to?

The more I've thought about this, the less I like my considered config. Buildfarm is exiting here because you have specified an endpoint that it cannot reach to perform any...

Distilling out the actionable parts of this issue: - Retries and timeouts for redis operations should be configurable - A threshold/immediate state transition to 'backplane inaccessible' from all consumption points...