Allow setting custom behavior when rate limit exceeded

[worst]

While rescheduling a job to occur later might be desirable for many situations, there are others where it's not.

For example, I have a job that is rate limited on a per-worker machine basis.

E.g., each machine is only allowed to process 1,000 jobs an hour, however I still want to process jobs as fast as possible.

Thus, I want to immediately add throttled jobs back to the job queue to allow another machine (which hasn't exceeded its threshold) to process it.

NB1: There is obviously a race condition in this example, but in practice it ensures that jobs are retried immediately, independent of the rate limit period, which is important for my use case. Note that a race condition also exists with the existing behavior (I think) because the job is being retried after a static period. Let's say that a job is limited to 1,000 executions per hour. 1 job was executed at :00 while another 1,000 were executed at :59. This will result in one of the jobs queued at :59 being retried in 60 minutes. If this pattern (1 job at :00, 1,000 at :59) continues ad infinitum, some throttled jobs might never be processed (or processed WAYYYYYYYY after they were submitted). At least I think that's how it would work...

To execute non-default behavior, simply specify a proc that takes up to 4 arguments (the period/delay of the rate limiter, the worker that's executing the job, the message payload, and the queue the job came from) in the :exceeded key in the worker class's throttler options.

NB2: The rate limiter now always passes these arguments to the specified exceeded block, so, while you can use a lambda instead of a proc, if you do so the lambda has to accept all 4 parameters. At least I think it does :)

[jayelkaake]

+1 would love to see this merged