Multiple Worker Server Environment

[mubbashar]

Hi, Thanks for this awesome package. Can you tell me how this handle scheduler command when there are multiple web server are running. Is it only run on one worker server Or the the scheduler will run on every worker server and this will cause duplicate message in the queue ?

thanks!

[dusterio]

@mubbashar every command will only be executed once - by one of the worker instances, the one that pops the job from the queue first (even though technically with AWS the queue comes to you as a POST request). There will be no duplicates

[mubbashar]

thanks for clarification. i was afraid if worker run schedule command then there will be duplicates because every worker will have their own cron.yaml file and every one call /worker/schedule every minute

[reganjohnson]

@dusterio We are considering this package. Could you clarify your point regarding the SCHEDULER and multiple workers?

We have multiple jobs setup that get executed at the same time (ie ->hourly()). Does the scheduler parse the commands that need to be run and push them to the queue as individual items or does the worker receiving the request process all jobs that evaluate true for the given cron statement (which could be many jobs).

[dusterio]

@reganjohnson one scheduled command (eg. hourly()) will only be passed and processed by one worker - whichever worker pulls from the queue first

[poisa]

For future reference: cron.yaml is only created in the leader instance in you aws workers. So if you have 10 instances, only ONE will run the crons. Technically one instance will create an SQS message which goes into the queue and is processed by any other instance that picks it up. If your queue is super busy, then this cron message might not be processed in time.

https://docs.aws.amazon.com/elasticbeanstalk/latest/dg/using-features-managing-env-tiers.html#worker-periodictasks

[arianacosta]

From that @poisa 's link regarding jobs defined in cron.yml:

Elastic Beanstalk uses leader election to determine which instance in your worker environment queues the periodic task. Each instance attempts to become leader by writing to an Amazon DynamoDB table. The first instance that succeeds is the leader, and must continue to write to the table to maintain leader status. If the leader goes out of service, another instance quickly takes its place.

[saleoid]

How I can process jobs in multiple queues with this. Any suggestions would be great?