What is the best way to ensure job execution rate?

[pss4281]

Hello,

got a question here. Let's suppose there is a use case like this:

• multi-instance job runners (sharing the same db)
• all jobs talk to an external service that caps requests rate (X reqs/second, if we exceed this - request fails)

so the question is how to insure proper rate (so that we don't make more than allowed number of requests/second i.e. we don't run more than allowed number of jobs/second) and keep the efficiency at a good level (e.g. if 10 req/second is allowed we do 10 and not 1 or 2)

I assume one of the options is to set each job's after field depending on prev. job's after? Any better way to tackle this?

thanks!

[vsivsi]

Hi, thanks for the question. job-collection is not a "real-time" system, in that it only makes a "best effort" to run/repeat/retry jobs when you ask them to. In your case you would like to maximize throughput among multiple workers without exceeding some hard rate-limit threshold. job-collection doesn't have a "built-in" way to do this.

What you need is some type of external sync mechanism that your workers can all access to get "rate permission" to access the external service.

It could be something as simple as a Meteor method call to the server that accesses an in-memory state variable that is set to (say) 10, once per second, by a Meteor.setInterval() callback. Each call to the method would decrement the state variable and return true if it is > 0, or false otherwise. Because node.js is single threaded, no fancy synchronization is required when accessing that variable on the server.

[pss4281]

@vsivsi thanks for a prompt reply! the setup you described (countdown quotas variable that gets refreshed every second) might work! I'm curious about how would the jobWorker skip it's turn if variable is === 0? We can fail the job in that case and hope that it will re-run in next second. But hope is kinda a bad thing to develop around :)

appreciate your answers

[vsivsi]

I wouldn't have the worker fail. I'd simply sleep it (with setTimeout()) for some amount of time (500ms?) and have it try again when it wakes up.

[pss4281]

thanks again @vsivsi this seems to be workable. Just need to fine-tune this sleep timeout to ensure they don't wait unnecessarily long.