Handling large number of topic subscriptions

[aschmahmann]

Currently if a node subscribes to enough topics such that the RPC that is used to send topic subscription announcements exceeds the maximum message size (defaults to 1MiB) then we'll silently fail and stop announcing any of our topics to newly connected peers.

We should more gracefully error, or inform the user, when they've exceeded the maximum number of topics they can subscribe to and additionally it'd be nice to have a mechanism in place to allow for a larger number of subscriptions.

• [ ] (Beginner) Error when exceeding maximum topic limit
  • Easiest: Just check if the (hello packet) message size is too large and log an error
  • More correct: When subscribing to a new topic check if the new topic would force the hello packet to exceed the maximum size and if so return an error instead of allowing the topic to be subscribed to.
• [ ] (Expert) Allow sending more topics than fit in a single message

[Stebalien]

Ideally, we'd be able to send a HAMT or a bloom filter of topics.

[BigLep]

@iand is working on this

[BigLep]

Beginner issue is to just get a proper error message.
This is back in the backlog.

[BigLep]

@aarshkshah1992 : for your rotation can you please take the error message at the minimum so that if this happens on Gateway we know about it?

[samlaf]

Is there a rough estimate of how many topics can be subscribed to using that 1MB limit?

[lthibault]

@samlaf I'd expect that to be hardware and network dependent.

[samlaf]

@lthibault Oh then I really have a bad understanding of this issue. I was assuming this was just saying that each subscription takes some amount of bytes, which need to be packed inside 1MB?

[aschmahmann]

@samlaf it's been a while since I wrote this issue and poked around in this codebase, but I think should be something like 1MiB/(sum of your topic name byte lengths + some overhead) and as a result it's a function of your topic names. i.e. you're encoding this https://github.com/libp2p/go-libp2p-pubsub/blob/d13e24ddc9f288557732c9c02b143c09b86c6778/pb/rpc.proto#L5-L15 and sending it over the wire and it's got to be under the max size.

[lthibault]

@samlaf As I understand it, the 1MB default limit pertains to the maximum size of messages transported over a pubsub topic. The maximum number of topics is therefore going to minimally depend on (1) network characteristics (2) available memory/cpu/etc of the machines, and (3) message transmission rate.