hollywood icon indicating copy to clipboard operation
hollywood copied to clipboard

Retry message failures

Open koola opened this issue 10 months ago • 3 comments

This PR is a continuation of #181 . It introduces two new options WithRetries and WithMaxRetries and aims to retry message failures on actor panics.

The retry logic is as follows;

pid := e.SpawnFunc(func(c *Context) {}, "foo", WithRetries(2), WithMaxRetries(2), WithMaxRestarts(1))

  1. Each message that panics is retried up to two times per message WithRetries(2).
  2. On the second retry, an ActorUnprocessableMessageEvent is broadcast to the eventstream containing the message and the message within the buffer is dropped.
  3. Following two consecutive message retries (4 retries), max retries are reached `WithMaxRetries(2).
  4. The actor is restarted to restore actor state and the messages left in buffer are resumed except those that were unprocessable.
  5. Retries are reset, but assume another two consecutive retries and restart. Max restarts are reached and the actor terminates and cleans up.

Using without retry;

pid := e.SpawnFunc(func(c *Context) {}, "foo", WithRetries(0), WithMaxRestarts(1))

  1. Each message that panics is dropped and actor is restarted as per original logic.

Included in this PR is a test that covers the full retry logic as well as amended tests.

All other tests pass.

koola avatar Feb 18 '25 09:02 koola