[Feat.Req] Pubsub network sharding

[hopeyen]

Problem statement On the relay network, all nodes should eventually receive all messages, which might cause too much work for certain nodes. To alleviate network traffic from participating nodes, we can utilize pubsub topic sharding. Waku propose potential technical solutions with RLN + Autosharding + message size limit + gossipsub.

Current available solution of static sharding may provide an immediate solution.

We should only work on this if the network performance is a proven issue.

Expectation proposal

• [ ] Determine the content topic distributions, select a fixed number of static shards.
• [ ] Provide a many-to-one hash function that takes any content topic and output exactly one shard.
• [ ] The radio nodes finds the relevant shard using the hash function and relay_subscribe to the specific shards.
• [ ] Identify the correct shards for the content topic when sending a message
• [ ] Update bootstrap nodes to relay new shards

example sharding: If we split the network into 9 shards, assume a uniform distribution of deployment representations, then for each deployment, we can simply let corresponding shard to be graphcast_mainnet_[deployment.bytes32().to_int() % 9]

• for a radio, upon generating and updating content topics, also generate the corresponding shards and subscribe to them
• when a radio sends a message, make sure the correct shard is used for publishing
• hosted nodes should each subscribe to 3 pubsub topics as their shards, but we should expect they are only used for bootstrapping

Alternative considerations

• RLN-relay requires a registration smart contract
• Autosharding is undergoing development by Waku team
• Once we determine the sharding scheme, we must be aware of the immigration cost when we update the scheme while the radios migrate/updates; or we should consider starting with a scheme that's easily backwards compatible.

Additional context Network sharding: https://rfc.vac.dev/spec/51/ Relevant comment: https://github.com/graphops/graphcast-sdk/issues/295#issuecomment-1778745913