semian
semian copied to clipboard
Throttle bulkhead from the circuit breaker on state transition
What
This PR attempts to address #244. It throttles the number of bulkhead tickets to 1 when the circuit opens, throttles to success_threshold
when the circuit moves to half_open
, and removes the throttle when the circuit finally closes.
Why
When the circuit is in an open state, we still allow all the workers that can acquire tickets to attempt the transition to half_open
. When the circuit is half open, we allow all those workers to attempt the transition to closed
, which can sap much needed capacity during a sustained outage.
Note: This PR is still a work in progress, and there are several corner cases that need to be addressed. I'm looking for a high-level review of the approach.
Issues
- It's a little gross to couple the bulkhead and circuit breaker in this way, but I'm not sure how else to approach it.
- Not sure how
SEM_UNDO
works when one process applies the throttle, and another removes it. Or what happens when the process that applied the throttle terminates. Needs to be thought through more.
cc: @Shopify/servcomm
Have you had any success with semsim?
No, I don't think this works at all...
SI_SEM_CONFIGURED_TICKETS
is basically a bookkeeping value. The real value we want to modify is SI_SEM_TICKETS
.
The problem is that at the point when we most likely want to reduce SI_SEM_TICKETS
to 1, it's already close to zero. If we attempt to do a -throttle
operation, that worker would block until the rest of the workers timed out on their requests. Given that we know we're opening the circuit, it would be awesome if we could terminate those requests immediately, but my brief research into this suggests that we'd have to take down the worker with a SIGTERM
which is likely a very expensive operation.