phoenixframework
Communication method for new topologies
Is there any preferred method of communication between processes? In particular using the underlying Erlang Node connections vs implementing something using gen_tcp or equivalent.
In general, from what I've heard in "battle stories" it might be advantageous to use the erlang distribution as a control plane and have separate tcp connections for actually sending data.
Right now big messages may clog the pipe and stall the heartbeat messages leading to a disconnection. At Lambda Days Peer (from GRISP) said, though, that they are looking into fixing this one.
Another consideration is that things like partisan don't use the default message sending and implement a completely separate layer for sending messages across processes (there might be even multiple "channels" between nodes). If we want to leverage this, I think we should do some sort of interface for delivering a message instead of using the built-in send.
The topology handles the remote pid communication. So our default disterl topology will use send, but others can implement their own mechanisms.
In Firenest.Topology we have the following definition: @callback send(t, node, name, message :: term) :: :ok | {:error, term}
But in the case of using something akin to gen_tcp, wouldn't the node have to be an IP? At most, we can assume the topology can internally map nodes to IP addresses or sockets.
The topology by definition would be responsible for translating the node name to something meaningful for its transport. The most trivial example could be using DNS A records to get an IP in the case of using :gen_tcp - in fact I think that's more or less how disterl does things today when you get down to the meat of it.
