Friedrich Gonzalez

> Sorry, I don't know what "mixin" is in this context. the cortex mixin contains dashboards and alerts, you can find it the latest in https://github.com/cortexproject/cortex-jsonnet/releases

> https://github.com/cortexproject/cortex-jsonnet/blob/main/cortex-mixin/dashboards/object-store.libsonnet ^ look in there

thanks for sharing, looks like you don't have that many requests, to be honest. The ones that are concerning are the querier and store-gateway errors. To reduce queries to block-storage...

Share ``` sum(rate(cortex_request_duration_seconds_bucket{container="distributor"}[5m])) by (route,status_code) ``` If you can

@jakubgs can you share your distributor config too? thanks

``` # Big queries need bigger message size. grpc_server_max_recv_msg_size: 104857600 grpc_server_max_send_msg_size: 16777216 # Bump gRPC concurrency to avoid delays. grpc_server_max_concurrent_streams: 1000 ``` I have never changed that. I ran clusters...

It's needed for ingesters though https://github.com/cortexproject/cortex-jsonnet/blob/main/cortex/ingester.libsonnet#L34-L36

From your message 10.10.0.214:9095,10.10.0.212:9095,10.10.0.211:9095,10.10.0.218:9095,10.10.0.213:9095 are the problematic ingesters. it might be worth checking if it is always the same ingesters in the message

Historically this was just a flag (and still is) ``` -ring.store ``` Which has nothing to do with ingester. But is at least more neutral. This then became ingester_config in...

Single company governed oss projects are an oxymoron. History tells they are bound to cause user pain and unneeded migrations in the long run. CNCF is an umbrella for users...