George Gensure

@zpzjzj can you rebase and revalidate tests with this?

https://github.com/werkt/bazel-buildfarm/tree/transform-token-timeout has the beginning of a failsafe for this condition. The transform tokens are exhausted - we cannot put any more onto its fixed size queue for tracking outstanding movement...

I don't believe this could still be happening now that we're switched to bazel 7. Feel free to reopen if present.

When bazel is stuck, what does jstack say about its currently running stacks? The connection reset is out of the control of buildfarm, but as you've indicated, probably caused by...

All of those 'error checking for' lines should at least have a failure stack trace printout: where is your throwable log data ending up?

> I met this problem before. The phenomenon is that `findMissingBlobs` will timeout. This is due to the worker cas "charge" is a synchronized method and it takes too much...

I'm on board with that, but to prove that that is where the resource (threads) exhaustion is, I'd need to see that jstack (or the failure to jstack due to...

Since we're in that function, and not in expireEntry waiting for a result (for this trace) I expect that we're waiting on the this lock as proposed. That means that...

Why are the evals being called so often compared to the former queue behaviors?

Is this resolved now?