Ankit Goel

Logs from first alertmanager were posted above. I had skipped some log lines related to previous resolved alerts(IngesterMemoryUsageAlert1 & IngesterMemoryUsageAlert2). The following are the logs for the other two alertmanagers....

Tried disabling sharding and ran a single replica of alertmanager. The issue is still happening.

@weeco Thoughts about this?

Thanks @weeco for the reply. I will go ahead and fork it :)

I am facing this same issue. When I run queries with a long duration i.e. queries >= 2 days, the store-gateways become unhealthy and the queries fail with the following...

We're running ingesters with -ingester.unregister-on-shutdown=true and -distributor.extend-writes=true. And we are still seeing the same behavior, deployment is stuck because one ingester is leaving.