reliability: option to safe guard against cardinality explosion

[eightnoteight]

trafficstars

prometheus doesn't support cardinality explosion by default, so when applications send a lot of metrics by mistake both statsd exporter and prometheus can get overwhelmed and lead to statsd exporter crashing due to memory or prometheus ingesting thousands of series and crashing or slow ingestion + slow queries.

So essentially I'm proposing a cli option to only maintain first N metric series received. and the rest to be simply dropped. Any thoughts on this area?

[matthiasr]

Hmm. I like the idea, but I am not sure how to make it "good" for users. The statsd protocol does not let us "reject" metrics so I am worried about silently dropping data. Combined with TTLs expiring old metrics, it could be very hard to understand what is going on. How would you like to get insight when this limit is hit?

[fpetkovski]

How would you like to get insight when this limit is hit?

I think this could potentially be solved by emitting a metric when the limit is hit and outputting an error log.