Stream oriented instead of peer oriented

[Stebalien]

Routing is currently peer oriented but we can probably make it significantly more robust by making it stream oriented. That is, completely forget about peer IDs.

Instead, when we receive a stream from a peer:

1. Read off subscriptions to topics.
2. Send messages on those topics back along the same stream.

Separately, when we notice that we have a connection to a peer, maintain at most one open stream to that peer where we:

1. Send subscriptions to topics.
2. Receive messages on those topics.

This makes handling multiple connections/streams pretty trivial.

---

Problems with the current system:

• We have to keep some concept of "are we connected".
• We can keep a peers subscription state even if they restart and lose it (they can disconnect and reconnect before we notice).
• We can process subscription additions/removals out-of-order if they come in on different streams.

[Stebalien]

Actually, we don't have to send messages back on the same stream. While I'd prefer to do that, we can also just send them back on our outbound stream.

The important part here is tracking our peer's streams independently.

[vyzo]

In principle it is more correct to use a stream oriented approach as it avoids a lot of pitfalls with disconnects, but it is a rather complex change.

[Nashatyrev]

Hey @vyzo @Stebalien ! Did I get it correctly, that pubsub communication between 2 peers is kind of 'half-duplex' in terms of streams? I.e. each peer opens a write-only stream and 2 -side communication requires 2 streams? If this is correct then can you please explain the rationale behind this solution? Thank you!

P.S. some discussion is here: https://discuss.libp2p.io/t/gossip-questions/257/6

[Stebalien]

To copy from that thread,

TL;DR: It makes it simpler. In libp2p, we generally use one of two patterns:

1. Request based. In this case, we open a duplex stream, send a request, and receive a response.
2. Message based. In this case, we open a stream and send messages to the peer. We don't expect responses to these messages.

In both cases, the peer opening the stream controls the stream.

Pubsub falls into the second category. We aren't sending requests, so we aren't expecting replies.

---

On the other hand, in message-protocols like this, we could just do both. That is, when we open a stream for sending, we could also spin up a listener on that stream to handle incoming messages. That way, the other side would get to decide what they want to do. The upside is that this would support non-multiplexed connections. But the rule "the stream opener controls the stream" is just a lot simpler and we don't expect the non-multiplexed case to be all that common.

[Nashatyrev]

@Stebalien thanks for detailed explanation! I think I understand your point.

However gossip appears to me like something between pattern 1 and pattern 2. Because there are IHAVE/IWANT messages which fall rather to the pattern 1. I've implemented the gossip with 2 streams and this approach introduces a number of edge cases like inbound stream is already open, but outbound is not yet. When one stream is closed, what to do with another one.

What I finally did is I effectively muxed those 2 streams into a 'virtual' single stream. That looks strange as we already have full-duplex stream capability

[Stebalien]

I'm talking about the request/reply pattern:

1. Send a request.
2. Wait for a reply.
3. Return the reply.

E.g., HTTP.

The key part here is actively blocking for a response. However, with gossipsub, IWANT doesn't result in a message being sent back on that same stream. Instead, it's just asking the other side to retransmit that message to us on the usual channel (when it gets around to it). It could have been implemented as a direct request/response, but that would have been a very different protocol.

What I finally did is I effectively muxed those 2 streams into a 'virtual' single stream. That looks strange as we already have full-duplex stream capability

You shouldn't need to do that. The protocol is designed to be message oriented so you should be able to model everything as sending and receiving messages. Really, you could open a new stream per message and receive messages on multiple streams.