operator
operator copied to clipboard
VictoriaMetrics
VMAlert should have remoteWriteSettings and remoteRead headers support
We use M3 as a persistent multi-tenant storage, where tenants are implemented by using some specific headers to filter metrics by labels or change metric labels before writing to the storage, for example for queries:
M3-Restrict-By-Tags-JSON: '{"match":[{"name":"remotewrite_source","type":"EQUAL","value":"zone1"}],"strip":["remotewrite_source"]}'
VMAgent supports remoteWriteSettings, so we can add additional labels before sending metrics to the remoteWrite storage. VMAlert does not, so for M3 it manages alert/rule states globally. We need to setup an additional proxy to add headers on read/write operations, which is not very flexible for multiple VMAlert instances.
If it's helpful, we can make changes by our team and submit them for review.
Hello, possible work-around - specify extra_label at remoteWrite.url:
kind: VMAlert
spec:
remoteWrite:
url: "http://vmsingle-example-vmsingle-pvc.default.svc:8429?extra_label=some-label=value"
Unfortunately it didn't work. VMAlert sections for remoteRead/remoteWrite:
remoteWrite:
url: "http://m3coordinator.example.com/api/v1/prom/remote/write?extra_label=remotewrite_source=test"
concurrency: 4
remoteRead:
url: "http://m3query.example.com?extra_label=remotewrite_source=test"
When querying ALERTS and ALERTS_FOR_STATE from m3query, remotewrite_source tags are missing from labels:
ALERTS{alertgroup="test-alerts", alertname="vmalert always-on test alert", alertstate="firing", job="<no value>", severity="test"}
ALERTS_FOR_STATE{alertgroup="test-alerts", alertname="vmalert always-on test alert", job="<no value>", severity="test"}
Found solution for remoteWrtie in the documentation: externalLabels, it works, but I'm not sure, that remoteRead uses those labels to limit alert requests.
Checked in the code, externalLabels is loaded at startup in the manager and used in rr/rw. Also, checked Alertmanager, multiple instances of VMAlert using different alerts from the remoteRead, filtering it by externalLabels. The current issue is no longer relevant, externalLabels allows us to use similar functionality to m3 header filters, thanks!
Unfortunately didn't test this for vmrule, it doesn't work, VMAlert queries metrics globally and writes rule results to global tenant.