Migrate discovery to an implementation of the SWIM protocol

[avishnyak]

I would like to propose that we migrate the service discovery to using the SWIM protocol instead of what is implemented currently. There are two advantages that I can see from this:

1. The current implementation is very chatty and depends on broadcast being allowed on a network which is not the case in most hosted environments (e.g. AWS, Azure, colos). Using redis can be a workaround for those environments but that has the issue of services dying unexpectedly and not updating their status in Redis.
2. The SWIM protocol can send both service info and keep-alive together making it a bit more efficient

Something like this: https://github.com/mrhooray/swim-js

[ericholiveira]

We can implement another discovery protocol. But just one info about 1. 1- We do not use redis to detect services dying (neither broadcast), we do it using primus (the transport layer), and this way we can also handle reconnection attempts (imagine a busy network) where a service is not dead but is unavailable for some time. The implementation of this other protocol can be really trick, but as long as we still have the security of the transport handling this things we can support this other protocol :)

[mchavarriae]

The problem with broadcast is when we are working in the same network with others developers, What do you recommend in that scenario?

[ericholiveira]

@mchavarriae You have 2 options: 1 - At least in your local development use redis as publisher (instead the default broadcast) this way you can isolate your communication per redis instance (so you just have to install it in each dev machine) 2- Turn off cluster while in development and run everything in the same process...