cortexproject
Higher latency on write and read path while rolling out ingesters (-ingester.unregister-on-shutdown=false)
Describe the bug
We're running ingesters with -ingester.unregister-on-shutdown=false and -distributor.extend-writes=false. This means that, while rolling out ingesters 1 by 1, the restarting ingester is left in the ring in the LEAVING state.
We've observed that, while rolling out ingesters with such configuration, the latency is significantly higher both on write and read path.
Expected behavior Ideally, no impact on latency when rolling out ingesters.
Storage Engine
- [x] Blocks
- [ ] Chunks
One theory is that the higher latency is given by the fact that we have to wait for the slowest ingester (because of the quorum) if there already 1 unhealthy ingester because of the LEAVING state. On the contrary, if there's no ingester in LEAVING state, we don't have to wait for the slowest ingester in the cluster.
We're running ingesters with -ingester.unregister-on-shutdown=true and -distributor.extend-writes=true. And we are still seeing the same behavior, deployment is stuck because one ingester is leaving.
