Monitor for dead consumers

[John-Connolly]

Currently SwiftQ does not monitor the processing queue for dead tasks. If a consumer crashes it will only be requeued on restart. SwiftQ uses per consumer processing queues consequently it is possible for tasks to get abandoned if that consumer is never restarted.

Possible Solutions

1. Keep track of every time a consumer pops a task and periodically monitor for timeouts.

2. Have task specific timeouts using the Redis SETEX command.

Monitoring

Every consumer will have to monitor every processing queue.