Define a better `Message` protocol

[greymistcube]

Still learning more about messaging patterns and discovering mounds of complexities required for a reliable publisher-subscriber pattern implementation. Seems like the harder option mentioned in this comment is much harder than initially thought. 😑

Some thoughts on the matter, in no particular order:

• Moving to a publisher-subscriber pattern only moves the responsibility of message transfer more from the sender to the receiver.
• Implementing and coordinating two different models on two different ITransport implementations, namely NetMQTransport and TcpTransport, requires a total of four different "chunks" to manage.
• Although ITransport is made to be business logic agnostic as much as possible, the way it is described and two implementations assume the usage of mixture of both a request-response and a publisher-subscriber pattern.
• Although a publisher-subscriber pattern seems like a not-so-hard solution at first, it is repeatedly warned that designing a such system requires far more care than a simple request-response pattern.
  • This matter is compounded if we want to have a reliable publisher-subscriber pattern.
  • Back-chatter cannot be completely eliminated, meaning additional structure and protocol is needed to support a reliable implementation.
  • Not too long ago, in #1741, RoutingTable dependency was removed from ITransport, but it seems ITransport itself needs to manage, at least partially, a RoutingTable. This would require additional interfacing. Although this point also applies to a proper management of peers for a request-response pattern, but a publisher-subscriber pattern seems to require a tighter coupling of ITransport and RoutingTable.

While working on #1845, I tried to leave the assumed Message protocol as is, as a pure request-response pattern seemed rather unorthodox, as suggested by many already. Also because changing what is essentially an assumed Message protocol to such as requiring a reply to every Message sent is no small decision to make. However, this also proved to be far more difficult than expected. As additional implicit logit had to seep in to Swarm<T> for handling Messages and ITransport had to be left purposely ambiguous making implementations also differ from what ITransport "seem" to suggest.

Plan:

• Add a new Message protocol document.
• Add a new Acknowledge type to Message instead of using Pong.
• Update ITransport interface to match the protocol.

This is so that if we ever decide to use a publisher-subscriber pattern in the future, there is a clearer path in fixing the issue.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. Thank you for your contributions.