tcp-info eventsocket should not deadlock on non-responsive client

[stephen-soltesz]

The current eventsocket architecture makes it possible for slow or non-responsive sidecar eventsocket listeners to deadlock the tcp-info service. This is an obvious problem for allowing third-party sidecar listeners.

I have demonstrated this by running the example-eventsocket-client using:

docker-compose up
docker pause  tcp-info_example-eventsocket-client
while docker exec -it tcp-info_tcpinfo_1 wget -O /dev/null www.google.com ; do sleep .1 ; done
# Wait until tcp-info container logs stop...

References:

• the eventsocket channel buffer is finite - https://github.com/m-lab/tcp-info/blob/master/eventsocket/server.go#L188
• new events are added to the channel by blocking calls to FlowCreated / FlowClosed
• all clients are enumerated sequentially - https://github.com/m-lab/tcp-info/blob/master/eventsocket/server.go#L79-L112
• client writes may block indefinitely - https://github.com/m-lab/tcp-info/blob/master/eventsocket/server.go#L83

Suggested requirements of a solution:

• tcp-info should treat clients separately - one should not be able to deadlock all others
• tcp-info may drop events for non-responsive clients
• tcp-info may close connections to non-responsive clients