client re-connect

[bradrydzewski]

We should provide the ability to re-connect a stomp client, with backoff, when the connection is broken. Right now we see issues and panics when this happens.

how this impacts drone:

This is important for Drone because the agent connection may be interrupted, and when this happens, the broker immediately re-queues the job. We need to account for interruptions so that agents can re-connect immediately without adding the job to the queue.

alternate drone solution

Instead of the broker immediately re-queueing jobs we could add jobs to a dead-letter queue when the connection is interrupted and the ack not received. When the client re-connects it could try to pull the item off the dead-letter queue. We would need to see how other systems handle this before we proceed.

[bradrydzewski]

question: should an unexpected disconnect provide some sort of reconnection period before adding all pending acks back to the queue? food for thought ...