django-celery-beat icon indicating copy to clipboard operation
django-celery-beat copied to clipboard

Published 20 hours ago • verified publisher icon celery

CeleryBeat unable to load large number of tasks

Open nitishxp opened this issue 4 years ago • 4 comments

Summary:

Include a brief descrioption of the problem here, and fill out the version info below.

  • Celery Version: 5.1.2
  • Celery-Beat Version: 2.2.1

Exact steps to reproduce the issue:

  1. Load 1 million task in Periodic task
  2. Start celery beat it will be stucked forever

Detailed information

We can have multiple(in millions) periodic active tasks. Below screenshot shows that celery beat is stucked in loading tasks from db image

nitishxp avatar Oct 21 '21 07:10 nitishxp

I overwritten the DatabaseScheduler to have implemented threading in schedule parsing does it make sense?

image

nitishxp avatar Oct 22 '21 04:10 nitishxp

can you come with a PR with that? to have more inputs and eyes?

auvipy avatar Oct 27 '21 10:10 auvipy

I'm having the same problem with around 30k ClockedSchedule tasks. The celery beat process seems to be stuck for some time at a stable medium cpu usage, then it drops and sends some tasks and starts again.

Any clue about where the problem could be or possible solutions? I would be happy to collaborate

ktowen avatar Oct 09 '22 00:10 ktowen

can you please try this patch and see if that helps? https://github.com/celery/django-celery-beat/pull/478

auvipy avatar Oct 13 '22 12:10 auvipy