bazelbuild
Rewrite repository worker threads using message passing
I spent a truly baffling amount on this, but I hope this is now deadlock-free and could be proven as such by a mere mortal.
That said, my confidence is zero in this so before making grand statements, let's see what the presubmits say.
@Wyverald WDYT? This is the best I could do. It's not bad and I can almost keep the state in my head. I still hate Exchanger (but I have no better ideas) but at least now the interplay between the two threads seems to be almost understandable. I also ran the "bazel build --noenable_bzlmod configure_with_bazel_transitive:all in a loop" test (passed) and the test battery did also.
The log messages should probably be removed, but they provide a good insight into what I had to do to get this to work so I thought it'd be useful, at least in the first iteration of this third iteration on your "separate Thread" idea...
I tried to ingest the logic here but it's honestly extremely tricky (probably not a surprise to you) even with all the comments. I'll try to continue tomorrow, but meanwhile I can't help but wonder if there is a better way to do this ("better" as in more maintainable down the line). Ultimately, what we want isn't that complicated: 1) at any given point, at most one of the worker and the host is active, and they exchange some information when "switching"; 2) if the worker is interrupted at any point for any reason, the host needs to wait for its termination.
I'm tempted to try again with yet another approach: instead of using BlockingQueue#take(), use poll(1s) in a loop instead, and check for liveness of the worker thread. I seem to remember you were not a fan of this approach in initial code reviews, but can't remember what's bad about it -- it essentially allows us to "select" from multiple conditions. Before I spend any time on this, I'd like to hear your thoughts on it. (Either way, I'll continue trying to understand this PR tomorrow...) (EDIT: I ended up using a semaphore which turned out great. See the original PR!)
