Ruler should be protected against high-cardinality output

[bboreham]

Suppose someone creates a rule, either recording rule or alert, that generates 100,000 output series.

Currently, all series will be sent in one request, which will hit the distributor rate-limit (defaults to 50,000 burst size) and be dropped.

This creates problems:

• ruler consumes a lot of CPU and memory
• it's hard for the operator hard to figure out what happened. The log line (caller=manager.go:539 msg="rule sample appending failed" err="rpc error: code = Code(429) desc = ingestion rate limit (50000) exceeded while adding 100000 samples") doesn't include the tenant ID
• it's completely invisible to the user of the tenant

I'm thinking ruler should cap the size of its output, and generate some signal (a synthetic series, perhaps?) that can be used to know when the cap was hit.

If we want to handle outputs from rules in the hundreds of thousands, we should batch them up so they don't choke the distributor.

The channel to alertmanager is also limited: caller=notifier.go:371 msg="Alert batch larger than queue capacity, dropping alerts" num_dropped=30973

[jtlisi]

I agree! The implementation is going to be a bit tricky I believe. I think we are going to have to write our own rule manager instead of using the prom upstream.

_{Sent with GitHawk}

[bboreham]

Perhaps this could be implemented via engineQueryFunc. Limits could be integrated with the existing limit/overrides system.

[bboreham]

Possibly addressed by prometheus/prometheus#9260

[jeromeinsf]

@krishnateja325 something you are looking at?

[krishnateja325]

yes, pulled-in https://github.com/prometheus/prometheus/pull/9260 and added support for limit field in this PR: https://github.com/cortexproject/cortex/pull/5528

[jeromeinsf]

/assign @krishnateja325