Gateway can cause Linux Kernel "TCP: Out of Memory" issue

[erulabs]

KubeSail Gateway can occasionally buffer too much memory in TCP sockets to packet-lossy or mis-behaving connected agents.

A real fix would be to update gateway handleSocket() to keep a highwaterMark for buffered packets (or rather, to track it), and pause the incoming socket when the destination socket is queuing.

Until then, a temporary fix is to dramatically increase the TCP buffer size in the linux kernel of Gateway servers:

net.core.netdev_max_backlog=30000
net.core.rmem_max=134217728
net.core.wmem_max=134217728
net.ipv4.tcp_max_syn_backlog=8192
net.ipv4.tcp_rmem=4096 87380 67108864
net.ipv4.tcp_wmem=4096 87380 67108864

Note that kubesail-agent users don't need to worry about any of this!