laravel-bref-bridge icon indicating copy to clipboard operation
laravel-bref-bridge copied to clipboard

Feature parity with traditional EC2 queueing

Open jszobody opened this issue 6 years ago • 0 comments

Ensure handling of queue jobs matches (or comes as close as possible) how a traditional queue worker would do it, running as a daemon on EC2.

https://laravel.com/docs/master/queues

Some of the items that come to mind:

  • [ ] Throttling # of queue jobs processed at a time. With a daemon we would control the number of queue workers. With lambda we want to make sure we have similar control so a boatload of queue jobs doesn't create unexpected load, or stampede a remote API, etc.

  • [ ] Retry queue jobs. Laravel keeps track of the number of queue job attempts in the serialized payload, need to replicate that.

  • [ ] Release job back to queue with a delay. A queue job might internally try/catch an operation, and call $this->release($delay) to trigger a retry with the delay. We use this for incremental backoff attempts for API calls to 3rd party services a lot.

  • [ ] Jobs that exceed max attempts should land in failed_jobs DB table: Just as with a traditional queue worker

  • [ ] Queued closures: ensure closure based queue jobs are handled properly (https://laravel.com/docs/master/queues#queueing-closures)

  • [ ] Job chaining: ensure chained queue job are dispatched appropriately, in order, and exception handling is the same (https://laravel.com/docs/master/queues#job-chaining)

  • [ ] Worker timeout (??): Laravel's queue worker has an optional timeout param, not sure if that's needed here or not.

jszobody avatar Apr 25 '19 15:04 jszobody