Worker consumes multiple tasks at the same time

[harryct229]

Description

Worker consumes multiple tasks at the same time when other jobs is not done yet.

• What is the current behavior? Worker consumes multiple tasks at the same time.

• What is the expected behavior? Worker consumes 1 task at the same time.

• Please tell us about your environment:

  • Version: 0.5.0
  • OS: [Linux]
  • Language: [ES6/7]

• Other information (e.g. detailed explanation, stacktraces, related issues, suggestions how to fix, links for us to have context, eg. stackoverflow, gitter, etc) So my worker is setup like this:

const testing_work = async (period) => {
  try {
    await someWork(period);
  } catch (err) {
    console.log(err);
    return err;
  }
  return null;
}

const worker = celery.createWorker(
  process.env.CELERY_BROKER, 
  process.env.CELERY_BACKEND,
  "testing_work"
);

worker.register("testing_work", async (period) => {
  try {
    await testing_work(period);
  } catch (err) {
    console.log(err);
    return err;
  }

  return null;
});
worker.start();

[actumn]

Hi. Thank you for reporting the bug.

As I see, you want to consumes 1 task at a time so that let the worker run synchronously. But unfortunately, we have not considered synchronous configuration.

I'll try to think about it in the future.

[harryct229]

Hi @actumn, sorry about the confusion. What I mean is do we have a limit of concurrence jobs. Because right now, the worker can consume all the resources while running all jobs at the same time. In Python celery, we have an option for --autoscale 3,2. Do we have this for JS version?

[alexio777]

it's not just a bug lack of synchronization (the ability to run only one worker) destroys the meaning of existence of celery in this universe