libp2p
feat(gossipsub): Add Message Batch Publishing in Gossipsub
Description
Adding a MessageBatch API to the libp2p::gossipsub to support batch publishing. This feature allows publishing a batch of related messages by prioritizing the transmission of one copy of each message to different peers before sending redundant copies.
Motivation
The Gossipsub protocol in rust-libp2p (implemented in the libp2p-gossipsub crate) currently allows publishing messages to topics individually, with each message being sent to connected peers as soon as it is published.
However, in scenarios with bandwidth-constrained peers or high message volumes, sending messages individually can lead to inefficiencies, such as redundant message propagation, dropped messages due to outbound queue limits, ..
One solution could be adding a MessageBatch mechanism to allow users to group related messages into a batch and publish them together (the implementation on go pubsub -> https://github.com/libp2p/go-libp2p-pubsub/pull/607).
This would optimize bandwidth usage by ensuring that a single copy of each unique message is sent to each peer before additional copies.
Requirements
-
MessageBatch Struct: A new struct to collect messages for batch publishing, with methods to add messages and trigger publication.
-
Batch Publishing Logic: A mechanism to prioritize sending one copy of each message to peers before sending duplicates, similar to the Go implementation’s rarest-first strategy.
-
Integration with Gossipsub: Extend the Gossipsub behaviour to handle batched messages, ensuring compatibility with existing message ID generation and peer outbound queues.
-
Configurable Queue Size: Recommend setting the peer outbound queue size to accommodate the expected number of batched messages, plus additional slack for gossip overhead.
Open questions
No response
Are you planning to do it yourself in a pull request?
Maybe
Hi, and thanks for the interest Toki.
Just to give some context, this was initiated on https://ethresear.ch/t/improving-das-performance-with-gossipsub-batch-publishing/21713 as an effort to try to improve message diffusion, specially in situations where the publishing peer is constrained on latency.
We have already started prototyping and testing on https://github.com/sigp/rust-libp2p/pull/571, we did it there as we want to properly test and assert that this is helpful first to lighthouse and then to rust-libp2p and the wider community before upstreaming it.
Is any of you @gitToki and @Richa-iitr users of gossipsub with interest in testing this feature? If so we are interested in users other than the Ethereum network.
If not I can help you find other issues to work on.
If not I can help you find other issues to work on.
I would like to take up on other issues!
Hi @jxs, thanks for the context. Yes, Csaba and I proposed this feature a long time ago in our DAS simulator, and now we are moving to real implementations. @gitToki is working with me in DAS research, and he would be interested in contributing to some of these features. I know the Lighthouse team is working on this feature, maybe @AgeManning can provide more details on the progress of it.
In any case, we are interested in contributing to rust-libp2p the features related to PeerDAS and DAS in general.
