statsd listener has no error handling, leading to catastrophic memory leaks

[andrewhamon]

The statsd client listener (a.k.a. zstream processor) has no error handling built in.

If there is an unhandled error, such as with buffer overflows as described in #123, then I'm pretty sure the fiber processing the stream will die (and in a rather silent way at that, I think fiber death by default just writes a stack trace to stdout)

This in turn will lead to the metrics queue filling up, since this library uses a bounded queue, rather than a dropping or sliding queue.

The default "async publishing" strategy forks a fiber for every publish that will block on offering the metric to the queue.

If the queue isn't draining (such is if the listener dies) then sending metrics will fork an unbound number of fibers until all memory is consumed.

I think the main takeaway here is that if publishing is async by default, then perhaps that shouldn't use a queue + stream at all, and instead fork and directly publish the metric in the same fiber.

Alternatively, a dropping or sliding queue should be used, so that offering to the queue will never block a fiber.