Enabling applications to remove/override messages in the message queue

[aschmahmann]

Discussed in #240 and #217

Some applications send messages which can render previous messages obsolete (e.g. I originally sent version 1, but now am sending version 2 of some data). In these cases it would be great if instead of the pubsub queue being backed up we could simply send less messages by not sending the obsolete messages.

Two example approaches to handle this might be:

1. Using topic options to handle shrinking the queue within pubsub
   • Add an topic option for a Merge(M1,M2 RPC) RPC function`
     • Could also use something simpler like FirstBetter(M1, M2) bool
   • Senders could have a per topic queue (relates to, but is not dependent on #210)
   • When adding to the topic queue run the merge function to determine the new message (or keep both messages)
   • Add the new message to the per topic queue
2. Exposing more data (e.g. via events/callbacks) outside of pubsub
   • Add a trigger to allow cancelling published messages
   • Add an event to learn about slow peers
   • If a peer is "too slow" then fire an event that causes the application to cancel the message
   • Add a callback for when messages would be dropped to figure out what to do with them (e.g. drop, keep)

[raulk]

@protolambda wanted this too