statsd-csharp-client
statsd-csharp-client copied to clipboard
lukevenediger
Batch & Pump metrics
So I see this issue is in the Future Improvements section of the Readme ("batch-and-pump - collecting stats and sending them out in a batch at regular intervals"), but I just wanted to check in on the progress.
I've already implemented a thread-safe Logger that does something very similar (using things like ConcurrentDictionary, lock, and Interlocked) that batches things up and "flushes" at regular intervals. I don't think it would be too difficult to use that as a starting point.
Also, does statsd.net server support multi-line metrics like Etsy's does? The flush process could be made even more efficient by sending packets with multiple metrics in each. See this code from the other "leading" C# StatsD Client for an example of safely sending multi-line metrics.
I think this would be nice, but I wonder what it would save when it comes to the udp client which has very low overhead/fire and forget.
Hey @niemyjski — totally agree when it comes to a fire-and-forget protocol such as UDP!
However, there are certain metrics that we want to reliably track 100% (or as close as possible) of the time. Due to the one-way nature of UDP, UDP will inevitably experience packet loss. This is fine for most things, but not all cases.
As such, we elected to use TCP for our most sensitive metrics to ensure delivery. As you well know, TCP is a much more heavy-weight protocol so we certainly cannot fire it off at the rate we fire UDP messages. So the idea is to batch up all sensitive metrics for a short period of time (say 5 seconds) and then send a single aggregated message to StatsD (which supports TCP).
Basically, it's a mini-implementation of StatsD that forwards messages to StatsD instead of Graphite! Of course I would have just installed a StatsD service on each of our servers if I had full say over the matter :smile: Oh well, such is the life of a lowly engineer!
Oh and I already implemented all of this using the other StatsD client since it already had a portion of the batching code done. It does exactly what I mention above, batches and flushes at regular intervals which reduces network traffic to negligible levels and ensures near 0 packet loss.
Do you think others would want this? If so, would you mind submitting a pr?