Severe performance issues

[arikfr]

We deployed fluentd to production using this plugin along with the out_redshift plugin.

Even during our initial benchmarks we saw that working with in_scribe gives far worse results than working with other input methods (like in_forward, which was giving 18kmsg/sec vs. 1kmsg/sec with in_scribe). But when we pushed real production traffic with all the plugins setup (during benchmark we used only in_scribe and out_file) it just couldn't handle the load (we're talking about ~300msg/sec).

It looks like the culprit is that all the message handling is happening on the same thread as the one that receives the Scribe messages and there is no actual use of Cool.io. So very often the processing gets delayed for some reason, the Scribe server will get a timeout and will stop sending data in until the retry period ends. But even then after a minute or so it dies again.

We worked around this issue by having in_scribe enqueue all the messages into a Queue and have another thread that will call Engine.emit on the messages in the queue. But this is sub optimal and far from being "production ready".