Local and global write queue limit in gossipsub

[arnetheduck]

Currently, we have a queue of 1024 pending writes in gossipsub, before we start disconnecting peers.

This is somewhat arbitrary and doesn't take into account large vs small writes - worst case with default settings, this can lead to a queue buildup of over 1000mb (1mb per gossip message).

Queueing happens in a number of scenarios:

• serving a peer that on average can keep up but needs helps with bursts
• serving a peer that on average can't keep up
• not enough bandwidth in general

Right now, this is not a problem because messages are typically small - it also doesn't take into account the importance of the messages being sent - ie a random gossip propagation is not critical.

An improvement would be

• count bytes instead of messages, per peer
• introduce a global limit for the gossipsub instance, to cap maximum memory usage
• degrade gracefully when either limit is hit
  • instead of disconnecting, we can drop messages and descore the peer