grafana
Standardise ring HTTP UI between Mimir components using it as client
Once https://github.com/grafana/mimir/pull/2957 will be merged, Mimir will have 3 rings used by multiple components:
- Ingesters ring
- Store-gateway ring
- Query-scheduler ring
Currently, all the rings UI is exposed by the components registering to that ring (e.g. /ingester/ring is exposed by ingesters), but only the ingesters ring UI is also exposed in the Mimir components using it as a client (i.e. /ingester/ring is also exposed by distributors, queriers, rulers), even if the HTTP API documentation page is not clear about it (it just mentions it's exposed by the ingester).
I think we should standardise on it. I propose to:
- Create the store-gateway and query-scheduler ring in
modules.goas we're doing for the ingesters ring, so that the same ring client instance can be passed around to multiple components when running in monolithic or read-write mode. This will implicitly fix this issue, because we would register the ring UI API too. - Improve the HTTP documentation, to be clear about which components expose
/ingester/ring,/store-gateway/ringand/query-scheduler/ring.
Thoughts?
Create the store-gateway and query-scheduler ring in modules.go as we're doing for the ingesters ring, so that the same ring client instance can be passed around to multiple components when running in monolithic mode. This will implicitly fix this issue, because we would register the ring UI API too.
How would it fix this issue? Do you suggest to register ring handler only by components reading the ring? (For query-scheduler that would be query-frontend and querier, which is opposite of what #2957 is doing.)
In the past only the components reading / watching the ring exposed HTTP endpoint for getting ring status. Recently our lifecyclers got the same capability. Perhaps we should switch it and only expose ring endpoint by lifecyclers? (So ingesters for ingester ring, store-gateways for their ring, etc.) I think that would be simple to understand, but unfortunately it's also a breaking change.
How would it fix this issue?
If we implement it in the same exact way of how it's done for the ingesters ring, it would register the UI API using the ring client (if available), falling back to the lifecycler one.
How would it fix this issue?
If we implement it in the same exact way of how it's done for the ingesters ring, it would register the UI API using the ring client (if available), falling back to the lifecycler one.
That would make sense. It has one drawback: if some component starts using the ring, then it will also expose the ring. This should trigger update of the documentation. In my proposed solution, there is 1:1 mapping between the endpoint and component.
In my proposed solution, there is 1:1 mapping between the endpoint and component.
That's true. The drawback, as you mentioned, is that we would introduce a breaking change and, I would add, we loose the ability to view the ingesters ring if all ingesters are unhealthy (and same for other rings). I still see some value exposing it from clients too, at the cost of potentially having a not perfectly synced documentation.
Should we expose rings from the client side when running in microservices mode as well?
