Inefficient TCP connections use by memberlist transport

[pracucci]

We use a custom transport for memberlist, based on TCP protocol. The main reason why we use TCP is being able to transfer messages which are bigger than the maximum payload of an UDP packet (typically, slightly less than 64KB).

Currently, the TCP transport is implemented in an inefficient way with regards to TCP connection establishment. For every single packet a node needs to transfer to another node, the implementations creates a new TCP connection, writes the packet and then close the connection. See: https://github.com/grafana/dskit/blob/ead3f9308bb7b413ce997182dd4d7c6e038bc68f/kv/memberlist/tcp_transport.go#L438

We should consider alternatives like:

• Pros/cons of keeping long-lived TCP connections between nodes, and multiplexing multiple packets over the same connection
• Using a mix of UDP and TCP, selecting the protocol based on the message size (in this case, TLS support wouldn't be available)