Fix head of line blocking in pubsub

[steeling]

With the pubsub used in the message broker, we allow for many publishers and many subscribers. In the current implementation, if a subscriber fails to read from its subscribed channel (or is slow to do so), it blocks publishers from being able to publish. In the worst case, a backed up subscriber can prevent certificates from being rotated, or block updates to proxies.

This can essentially take down a single osm-controller. We can degrade (more) gracefully by instead leveraging an unbounded queue when publishing, prior to reading publishing onto the

While we're at it, we should hide the internals of the pubsub mechanism from the broker's API calls, and simplify using channels. An API may look like:

func (b* Broker) PublishCertEvent(e Event, topics ...string) {
   ...
} 

// Named return params just to show what the unsub func is used for.
func (b* Broker SubscribeCertEvents(topics ...string) (stream ch <- Event, unsub func()) {
   ...
}

This still prevents updates from being published, but at least frees up routines that are doing the publishing. We should look into other mechanisms to further degrade gracefully. Some may include:

1. Removing/reducing usage of pubsub
2. Having a policy of no blocking calls on any routine that subscribes to a pubsub topic.
3. Use an unbounded queue on the subscriber end. If we do this, we may want to further investigate message coalescing.

[github-actions[bot]]

This issue will be closed due to a long period of inactivity. If you would like this issue to remain open then please comment or update.

[github-actions[bot]]

Issue closed due to inactivity.

[keithmattix]

Still valid

[github-actions[bot]]

This issue will be closed due to a long period of inactivity. If you would like this issue to remain open then please comment or update.

[github-actions[bot]]

Issue closed due to inactivity.