cortex
cortex copied to clipboard
cortexproject
Proposal: introduce read-only mode for ingesters
Recently we had to deal with a Cortex outage during which ingesters (running the blocks storage) were failing to compact head due to a in-memory corruption in TSDB (which has already been fixed in https://github.com/prometheus/prometheus/pull/7560). During the outage we had the need to stop ingesting samples on some ingesters, while keeping them running for the queriers in order to not loose series when querying, but unfortunately we haven't found any way to do it.
As a follow up action from this outage, I would like to propose to introduce in the ring the ability to mark an ingester as read-only. When manually marked as read-only, the ingester is ignored by distributors on the write path, while queries will continue query it.
Thoughts?
I think this can be useful for operations, and storing read-only flag to ring seems logical.
Implementation-wise, as ring is used by different components (store-gateway, compactor, HA), I'm wondering if we should use some more generic "tags" instead, so that each instance can be marked with list of strings (tags), understood by the components. For ingesters, one such tag can be "read-only", which ingesters and distributors would understand, and admin UI would simply show it as a string.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.
![stale[bot] avatar](https://avatars.githubusercontent.com/in/1724?v=4 "Published by a pub.dev verified publisher"){kind=small}
Today we had another incident where this feature would have been very useful.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.
This issue has been automatically marked as stale because it has not had any activity in the past 60 days. It will be closed in 15 days if no further activity occurs. Thank you for your contributions.