Dogstatsd Buffer Size 8k

[sschepens]

Is there a reason why Dogstatsd sets buffer size to 8k? Maximum UDP payload size is actually around 65k and should be ok, I can't find anywhere in statsd's source where they limit buffer size on the server side. The size of messages should be configured by the client and the server should just read the messages.

We're sending lots of metrics from our servers and we wanted to increase the packet size, but packets get cut because Dogstatsd is only reading 8k.

[remh]

8k should be more than enough for anyone!

More seriously, it's usually recommended to have the max buffer size set to a small power of 2 (see https://docs.python.org/2/library/socket.html#socket.socket.recv ).

That being said there is no reason for it not to be configurable. I'll add this to our backlog.

Thanks for the feedback.

[sschepens]

It's not really much considering statsd format is very verbose and a command gets sent for every counter o gauge reported. If i call count multiple times,no buffering occurs and all sends are triggered, this, plus long names and possibly lots of tags makes packets big, and an application could see a huge improvement in performance by using larger packets. Our application was running 7 servers at 50% CPU, I realized it was because we were logging lots of metrics, disabling datadog metrics dropped our servers to 2 with 30% CPU... I then proceeded to internally buffer counters and flush them at a regular period to reduce the amount of UDP sends, then I increased the packet size, but hit this issue, when theoretically i could use UDP packets of up to 64k payload. I ended up using 8k packets, and it seems ok, but I still do counter buffering.

[elijahandrews]

@sschepens Thanks for the feedback, we're looking into making the maximum buffer size in the agent's dogstatsd server configuration. You may also want to try sample rates, which will reduce the number of metrics you submit. See http://docs.datadoghq.com/guides/metrics/#sample-rates

[ebpitts]

I see #2804 was closed. Is there still a plan to address this?

[souravchandra]

Any updates on this ticket, whether this property is now configurable. We also face similar issue because of this.

[mbelang]

I just enabled DD_LOGS_CONFIG_K8S_CONTAINER_USE_FILE environment variable on k8s deployment and since then I have long longs that are now truncated and spans over 5 rows/entries. Is that related to the max packet size? Never saw that behavior before activating the option.