VictoriaMetrics
Urls in labels should be easy to find
Is your feature request related to a problem? Please describe. Sometimes after code refactoring labels will contain not what was expected since start. Once i got an urls in label "calling_service" instead of service name. And i got long url with get params in it. That was unexpected from both developer and admin side. Of cause that leads to large cardinality
Describe the solution you'd like The main idea is to get an list with blacklisted chars in label value. Than count if value matchs agains it. Probably that should not apply to each insert operation but can be applied to some sampling subset.
Describe alternatives you've considered that operation can be applied on metric relabling during scrape operation too. But i can't do anything with it excerpt drop label or even metric Also that operation can be done on select, but i really have no idea how enumerate all labels and find chars in it
Side note: it is possible to remove unsupported chars from label values with relabeling in vmagent and single-node VictoriaMetrics with replace_all action. See these docs for details.
For example, the following relabeling rule replaces all the chars outside a-zA-Z0-9 with _ in the calling_service label:
- action: replace_all
regex: "[^a-zA-Z0-9]"
replacement: "_"
source_labels: [calling_service]
target_label: calling_service
yep it is. but it will not fight agains carnality. Only replaces values. The idea is to find that we probably have misconfig and then find where and fix it. not hide.
VictoriaMetrics exposes labels with the maximum number of label values at labelValueCountByLabelName array at /api/v1/status/tsdb endpoint. Probably this endpoint could be used for quick detecting of high cardinality labels. VictoriaMetrics accepts optional date=YYYY-MM-DD and topN=42 args on this page. By default date equals to the current date, while topN equals to 10.
and again that was not so helpfull, i have several kubernetes clusters and of cause labels like id and pod and service has much more cardinality.
by the way topN looks great and probably will help diagnose problem
FYI, take a look at https://github.com/open-fresh/bomb-squad (thanks to Ren from slack community chat for the link).
Unfortunately it seems that the bomb squad project is dead. Would be nice if something similar is included in vm.
And more over should this issue have to be generalized to "fight against high carnality" or has separate value i'm not sure. For me this is a red flag that we have some issues for debug not the debug it self
Hi @freeseacher! Have you considered https://docs.victoriametrics.com/vmagent.html#cardinality-limiter for control over cardinality?
