yarpc-go
yarpc-go copied to clipboard
Bump grpc-go version; fixed grpc connection status mapper
Since grpc-go 1.41.0, failed connections are moved to IDLE state without connection retries. Reconnection attempt is happening either by direct call to Connect method, or with an attempt to make a request using given connection.
Yarpc-go library is built with an expectation of automatic reconnections done by grpc, and only grpc-go connections in READY state are used for outbound requests. READY grpc state is mapped to Available yarpc status, Connecting - to Connecting, all other (IDLE, TransientFailure, Shutdown) - to Unavailable.
Without the fix presented in this PR, eventually all grpc-go connections may be moved to IDLE state, which maps to Unavailable state from yarpc perspective. It creates a deadlock: yarpc waits for connection to be moved to READY state; meanwhile grpc keeps IDLE state and wait for the rpc call or explicit Connect call.
This PR introduces an explicit Connect call on connections that are moved to IDLE state. It forces grpc to try reconnection, which moves connection into Connecting state (for both grpc and yarpc). If connection succeed, state will be set to READY (yarpc: Available). If failed, new state is TransientFailure (yarpc: Unavailable), then IDLE (moved by grpc-go; yarpc: Unavailable; triggers a reconnection attempt one more time).
We don't have to implement reconnection backoff manually, because it's part of the reconnection logic in grpc library.