krill
krill copied to clipboard
Metric explosion for `krill_repo_size` and `krill_repo_objects` metrics
As a user I want to make high cardinality metrics optional so that it does not affect my prometheus instance.
We use krill in a test environment where we automatically create repositories for integration tests. This means this krill instance has a significant number of labels.
We noticed that the amount of metrics in the prometheus instance monitoring this krill instance exploded. When debugging what metrics had high cardinality, we found that the krill metrics were slowest:
Error fetching labels for krill_repo_objects: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_objects'
Error fetching labels for krill_repo_size: 504, message='Gateway Time-out', url='https://prometheus.internal/api/v1/series?match%5B%5D=krill_repo_size'
krill_repo_last_update also has a high number of publisher values