TCP device loses events through timeout proxy

[sheldonh]

trafficstars

When the TCP device is used to send events to Logstash through a proxy that uses timeouts instead of TCP keepalives, it is guaranteed to lose events. A quiet client that logs only once per timeout period will observe 50% event loss.

I ran into this with https://logit.io/. They front Logstash with a proxy that uses application layer idle timeouts instead of TCP keepalives. I'm chatting to them about it, but meanwhile, here is a demonstration of the problem:

https://github.com/xneelo/elk-spike

I've had real trouble navigating the inheritance shenanigans to figure out what's going on, but it seems like the TCP device doesn't interact well with Connectable's retry strategy.

[sheldonh]

One filthy hack would be to emit blank lines periodically. The json_lines codec ignores blank lines.