prometheus
Is there a reason why statsd exporter should not be run as a daemon set?
The only method mentioned in the documentation for how to run statsd exporter is as a sidecar. I wanted to know if this was because there are known issues with other methods of deploying statsd exporter, such as with a daemon set, or if it was because the documentation is focusing on getting users on the quickest path to getting running?
I would presume running as a daemonset is fine since statsd exporter is available as a component in Grafana Alloy which supports running as both a daemonset and a stateful set. I have a use case where running as a daemon set would be useful, and wanted to know if I am about to shoot myself in the foot before I do it.
Yes, there are behavior "known issues" with DaemonSets, same problems with using agents like Grafana Alloy and OTel.
One of the fatal flaws of push-based metrics systems is that the lifecycle of a metric definition is completely unknown.
How do you know when a process is restarted and a counter metric should be reset to zero? With a DaemonSet, this is quite difficult.
How do you plan to differentiate between difference instances of the same workload on the same node? What if you're not running Kubernetes, but some other orchestration? You now have to implement service discover in the protocol (OTel) or in the agent. This is much more complicated than Prometheus's centralized discovery.
The next problem is how do you expire metrics? If you have a metric you're no longer reporting, or if you add or remove a label/tag from a metric? With a DaemonSet/agent, you have a much more difficult time expiring for different instances.
By using the sidecar pattern, all of the above problems are solved since the lifecycle of the exporter matches the lifecycle of the Pod/instance.
This is why OTel's push design is a fundamentally flawed system and should be avoided. It follows too many of the design flaws of StatsD. This is the reason the statsd_exporter was created, not just to convert the protocol, but to solve the problems of the push model.
+1 to @SuperQ's concerns. That being said, if you have a sufficient need, and can handle the issues that arise from the daemonset pattern in your environment, nothing will prevent you from doing that, and I know some people run the exporter that way.
In our documentation, we're promoting the sidecar pattern because it is the best solution, and maybe not the most obvious one to someone coming from statsd/graphite (which is still the main target for this exporter). I don't want to confuse the messaging there by calling out the daemonset option; for anyone who digs deeper, let this issue stand as their starting point 😄
Using statsd as a daemonset is against mathematics. A metric cannot be counted in 2 places at the same time, or counted in one place and its previous value from the cache published in the other. The solution is a statefulset in which traffic is directed via statsrelays, but also in this architecture a rigid structure must be maintained, the number and order of instances must be permanent, and any change requires resetting the counters on all instances so that they "forget" what they had.
