Support using DaemonSets

[hifi]

It would be very useful to be able to run at least some of the components as DaemonSets instead of scaling them manually. We have dedicated nodes for vmstorage for example and a DaemonSet would allow automatically scaling when adding a new node.

If this is already possible I couldn't find it from the docs.

Thanks!

[f41gh7]

Hello, currently it's not possible and we don't have such plans.

I'd recommend to take look at affinity rules for pod scheduling. It's possible to schedule pod per node with it.

[zekker6]

Support of using DaemonSets might be helpful for vmagent and vmauth for cases of reducing inter-AZ traffic. For example, vmagent can scrape metrics only from targets within the same AZ (by using relabeling + propagation of properties via downward API) and than route metrics to the destination via vmauth running on the same node.

[f41gh7]

Support of using DaemonSets might be helpful for vmagent and vmauth for cases of reducing inter-AZ traffic. For example, vmagent can scrape metrics only from targets within the same AZ (by using relabeling + propagation of properties via downward API) and than route metrics to the destination via vmauth running on the same node.

It's already possible to get rid of cross-AZ traffic with labels. See https://tanmay-bhat.medium.com/reducing-inter-az-traffic-in-victoriametrics-with-zonekeeper-3bd7e1526796

Also I'd not recommend to run vmagent as daemonset if it uses non-local service-discovery. Like kubernetes_sd_configs. It may have significant impact on Kubernetes api-servers and probably lead to whole cluster downtime ( like chatGPT had).

[zekker6]

@f41gh7 There is another use-case where applications are pushing metrics. In this case apps could push to a local address and vmagent would push the data to the receiver after applying compression.

[f41gh7]

Okay, we agreed to implement this feature. However, it'll have some limitations.

Goals:

• provide seamless migration from deployment/statefulset to daemonset
• daemonset mode must be compatible with the most common scrape objects - VMPodScrape, VMServiceScrape
• daemonset mode must efficiently utilize Kubernetes api-server.

Limitations:

• HPA is not supported
• PDB is not supported
• Sharding is not supported
• storage could be hostPath or emptyDir
• VMNodeScrape, VMStaticScrape, VMScrapeConfig and VMProbeScrape are not supported

Viable Kubernetes api-server interaction optimizations:

• role: pod must have field selector:

 selectors:
 - role: pod
   field: spec.nodeName=%{NODE_NAME}

• VMServiceScrape must use endpointslices instead of endpoints
• VMServiceScrape with discoveryRole = service must be rejected.

[f41gh7]

Feature was included into v0.55.0 release