Advanced Exception Handling

[jeremydmiller]

Inspired by a project at work. What if the system is hosed so badly that no message could possibly be processed? Shut the listener down altogether, find some way to "know" when it's available again.

I've got some notes somewhere on possible implementation. Find those, and add them here.

[jeremydmiller]

Tasks

• [x] Gotta ensure that the listener Uri is set on incoming envelopes, that or attach the IListener
• [x] IListener.Stop()
• [x] IListener.Restart() on each
• [x] Add something to the compliance tests
• [x] Use an in memory scheduled job? Or make a timeout in the listener itself?
• [x] In compliance, check that all listeners say the status is Accepting on start up
• [x] Harden the listeners against possible failures when stopping

[jeremydmiller]

Vision

Renaming this and expanding it from the original idea of just having a manual circuit breaker. Here are the intended use cases:

• Make it easy to publish another message on certain exceptions -- cannot be coupled to cascading messages. Hooks are there, just need an example and extension method maybe
• Have an option for discarding messages -- Already exists
• Option to move messages to a different queue
• Cannot lose messages when the listening agent is shut down. Try to defer all in flight messages in the buffered receiver
• Allow users to modify an envelope on failures???
• Register "on message failure" policies? Might be easier.
• Need some tests on mixed error handling

Circuit Breaker / Kill Switch

Trip Conditions

1. On a certain exception condition, trip it off. This is for the case Jeremy hit years ago where a downstream system would be up, but completely invalid, so you wanted to just immediately shut down any additional processing.
2. If a certain percentage of messages in a time window fail, shut down the circuit

What to do when it trips off?

1. Send a message
2. Try to restart after a certain amount of time
3. Move all incoming messages to dead letter queue?
4. Shut down the listener?
5. Discard incoming messages
6. Rewind envelope attempts / reset attempts

[jeremydmiller]

Circuit Breaker Permutations

• [x] Stay open w/ Rabbit MQ + Inline receiver
• [x] Stay open w/ Rabbit MQ + buffered receiver
• [x] Stay open w/ Rabbit MQ + durable receiver
• [x] Shut and reopen w/ Rabbit MQ + Inline receiver
• [x] Shut and reopen w/ Rabbit MQ + buffered receiver
• [x] Shut and reopen w/ Rabbit MQ + durable receiver
• [x] Shut and reopen w/ Parallelized Rabbit MQ + Inline receiver
• [x] Shut and reopen w/ Parallelized Rabbit MQ + buffered receiver
• [x] Shut and reopen w/ Parallelized Rabbit MQ + durable receiver
• [ ] Shut and reopen w/ Pulsar + Inline receiver
• [ ] Shut and reopen w/ Pulsar + buffered receiver
• [ ] Shut and reopen w/ Pulsar + durable receiver

[jeremydmiller]

Punchlist

• [ ] Implement the AlsoDo
• [ ] Add recipe for sending a message as a response to a failure.
• [ ] Documentation