mimir
mimir copied to clipboard
grafana
Alerting on compactor falling behind
Hi, I have few questions regarding compactors scaling. We have a cluster with ~150M active series, 60 tenants. For few large tenants (one around 80M series, few with 5-20M series) we have two-level compaction (split-and-merge) with different split groups and merge shards count configuration. For small tenants (<8M series) we use single-level compaction (one or more shards, no splitting level/stage). Currently we have 72 compactor replicas (k8s req,lim 2 cpu, compaction_concurrency=2). It seems that this is far more than what is proposed in capacity planning documentation.
However, when we experiment with differnet setups, it's not easy for us to see/(alert on) if compactor is not falling behind. Because of that, we have created a small component mimir-bucket-exporter which exposes bucket metadata based on what is in the per-tenant bucket index. The aim was to be able to alert on case in which we would query blocks in storage for which compactions haven't finished yet.
Q1: How we can tell if compactor has already finished its job for given time period? We originally thought that we can just check if all expected shards exists for a certain block level, but target level seems to depend on tenant's configuration. Tenants with single level compactions finish at level 2 (for 2h blocks), while tenants with two-level compaction finish at level 3. Is there a way how to find out if compaction for given time range is done without knowing what is configuration for particular tenant?
Q1.1: Are blocks which result from split stage used for querying or are ignored by store-gw?
Q2: Is it possible to have dedicated set of compactors for 1 or more large tenants? We have had issues with missing bucket index when we tried to dedicate few compactor replicas to a single tenant. It seems that this only can be done safely with 1. a single all tenants compactor ring and 2. tenant-dedicated set of instances with different compactor ring (but haven't tried this and are not even sure if this is a valid approach).
Q3: Is it possible so have a dedicated set of compactors which will only generate bucket indexes and second set which will handle the compaction itself?
The motivation for Q3 is that we have tried to reconfigure compaction_interval from 1h, 15m. This had positive effect that it decreased delays before compactor picks a new job but it brings a lot of pressure on our S3.
@brendamuir @tacole02 This one talks about Alerting, but it also specifies it's in the mimir repo.
@david-vavra - hi David: quesiton -- the questions above seem directed toward engineering for answers. Can you be more specific on what changes you want made to the documentation?
Hi, @derek-cadzow , sorry for the delay.
It would be nice if Mimir mixin would contain alerts coverting situation where compactors are under-scaled, and thus are not able to compact blocks quickly enough until uncompactd blocks get hitted by queries. There are already some alerts which are covering various error-cases, but alert covering that Compactor is succesful in its primary function is missing. Documentation-wise, runbook for such scenario would definitely be helpful as well.
@zhehao-grafana could you please comment on whether this issue still requires documentation, and if so, provide a relative priority? Thanks!