django-celery-beat icon indicating copy to clipboard operation
django-celery-beat copied to clipboard

Published 20 hours ago • verified publisher icon celery

Django Celery Beat not sending tasks to Celery Worker

Open pisaia01 opened this issue 3 years ago • 17 comments
trafficstars

Summary:

Celery-beat not sending the tasks to celery worker for execution.

  • Celery Version: 5.2.3
  • Celery-Beat Version: 2.2.1

Exact steps to reproduce the issue:

  1. Start the Django project using docker-compose up -d
  2. Stop the Django project using docker-compose down
  3. Restart the Django project using docker-compose up -d

Detailed information

When I first start the Docker, all the tasks run without any issue. If I stop the docker (i.e. docker-compose down) and then restart the docker (i.e. docker-compose up), celery-beat does not send the tasks to the celery worker in order for them to get executed.

If I visit the Admin panel and disable the tasks and then re-enable them it starts working!

Also if I do not use the django_celery_beat.schedulers:DatabaseScheduler and allow celery beat to use the default scheduler it also works. Even though this method works, is not the best scenario since the tasks are not longer editable in Django admin panel

Is there a solution to this issue?

Postgresql is not part of the docker-compose. I have a native PostgreSQL 10.5 installation.

settings.py

CACHES = {
    "default": {
        "BACKEND": "django_redis.cache.RedisCache",
        "LOCATION": "redis://redis:6379",
        "OPTIONS": {
            "CLIENT_CLASS": "django_redis.client.DefaultClient",
            "CONNECTION_POOL_KWARGS":{"max_connections":50, "retry_on_timeout": True}
        }
    }
}

SESSION_ENGINE = "django.contrib.sessions.backends.cache"
SESSION_CACHE_ALIAS = "default"
CELERY_TIMEZONE = "Europe/Amsterdam"
CELERY_TASK_TRACK_STARTED = True
CELERY_TASK_TIME_LIMIT = 30 * 60
CELERY_BROKER_URL="redis://redis:6379"
CELERY_CACHE_BACKEND = 'default'
CELERY_BEAT_SCHEDULE = {
    "sendemails_task":{
        "task":"demoapp.tasks.testing",
        "schedule":crontab(minute='*/2')
    },
}

docker-compose.yml

services:
  redis:
    image: redis
    restart: unless-stopped
    expose:
      - 6379
  web:
    build:
      context: ./app
      dockerfile: Dockerfile.prod
    restart: unless-stopped
    command: gunicorn Project.wsgi:application --bind 0.0.0.0:8001 --workers=4 --preload
    volumes:
      - static_volume:/home/app/web/staticfiles
      - ./uploads/:/home/app/web/uploads/
    expose:
      - 8001
    env_file:
      - ./.env.prod
  celery:
    build:
      context: ./app
      dockerfile: Dockerfile.prod
    command: celery -A Project worker -l info
    volumes:
      - ./app/:/usr/src/app/
    env_file:
      - ./.env.prod
    depends_on:
      - web
      - redis
  celery-beat:
    build:
      context: ./app
      dockerfile: Dockerfile.prod
    command: celery -A Project beat -l debug --scheduler django_celery_beat.schedulers:DatabaseScheduler
    volumes:
      - ./app/:/usr/src/app/
    env_file:
      - ./.env.prod
    depends_on:
      - web
      - redis
      - celery
volumes:
  static_volume:

tasks.py

from __future__ import absolute_import, unicode_literals
from celery.utils.log import get_task_logger
from celery import shared_task
from django.core.cache import cache
import time
import traceback


@shared_task
def testing():
    print('test')

celery.py

from __future__ import absolute_import, unicode_literals

import os

from celery import Celery

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'Project.settings')

app = Celery('Project')
app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

celery beat log

[2022-01-31 16:24:56,138: DEBUG/MainProcess] beat: Ticking with max interval->5.00 seconds
[2022-01-31 16:24:56,217: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:01,223: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:25:01,223: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:25:01,241: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:06,260: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:11,282: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:16,303: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:21,433: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:26,451: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:31,472: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:36,491: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:41,511: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:46,533: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:51,615: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:25:56,636: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:01,654: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:06,669: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:11,692: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:16,710: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:21,733: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:26,754: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:31,770: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:36,949: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:41,971: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:46,989: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:52,010: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:26:57,030: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:02,051: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:07,073: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:12,093: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:17,115: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:22,137: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:27,155: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:32,176: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:37,208: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:42,228: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:47,245: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:52,266: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:27:57,288: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:02,293: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:28:02,294: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:28:02,395: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:07,413: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:12,431: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:17,478: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:22,499: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:27,522: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:32,545: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:37,564: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:42,585: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:47,602: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:52,614: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:28:57,629: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:02,653: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:07,991: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:13,011: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:18,029: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:23,097: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:28,115: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:33,135: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:38,153: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:43,167: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:48,184: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:53,202: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:29:58,222: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:03,238: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:08,258: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:13,270: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:18,288: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:23,305: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:28,360: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:33,382: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:38,403: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:43,424: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:48,440: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:53,459: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:30:58,490: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:03,496: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:31:03,496: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:31:03,510: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:08,572: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:13,592: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:18,616: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:23,637: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:28,660: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:33,682: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:38,719: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:43,822: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:48,840: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:53,857: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:31:58,877: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:03,898: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:08,919: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:13,940: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:18,959: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:24,064: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:29,087: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:34,106: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:39,127: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:44,145: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:49,166: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:54,184: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:32:59,207: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:04,344: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:09,366: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:14,383: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:19,400: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:24,418: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:29,437: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:34,459: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:39,477: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:44,499: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:49,522: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:54,542: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:33:59,563: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:04,568: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:34:04,568: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:34:04,580: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:09,598: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:14,621: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:19,642: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:24,662: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:29,683: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:34,702: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:39,722: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:44,741: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:49,759: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:54,778: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:34:59,800: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:04,819: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:09,838: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:14,860: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:19,982: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:25,001: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:30,018: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:35,039: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:40,058: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:45,078: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:50,098: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:35:55,118: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:00,140: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:05,159: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:10,181: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:15,195: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:20,213: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:25,233: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:30,338: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:35,360: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:40,382: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:45,403: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:50,425: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:36:55,487: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:00,507: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:05,511: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:37:05,511: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:37:05,523: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:10,541: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:15,568: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:20,585: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:25,607: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:30,694: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:35,713: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:40,734: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:45,796: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:50,814: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:37:55,834: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:00,854: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:05,874: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:10,895: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:15,913: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:20,935: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:25,955: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:30,975: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:35,996: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:41,014: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:46,033: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:51,052: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:38:56,068: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:01,087: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:06,101: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:11,122: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:16,176: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:21,193: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:26,214: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:31,235: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:36,251: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:41,266: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:46,282: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:51,299: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:39:56,319: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:01,338: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:06,343: DEBUG/MainProcess] beat: Synchronizing schedule...
[2022-01-31 16:40:06,343: DEBUG/MainProcess] Writing entries...
[2022-01-31 16:40:06,357: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:11,377: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:16,399: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:21,421: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:26,438: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:31,459: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:36,481: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:41,498: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.
[2022-01-31 16:40:46,514: DEBUG/MainProcess] beat: Waking up in 5.00 seconds.

pisaia01 avatar Jan 31 '22 14:01 pisaia01

Is there any news about this bug?

ilourt avatar May 30 '22 17:05 ilourt

@ilourt @pisaia01 We are seeing the same problem. With docker. First run works. On subsequent runs unless we disable/enable it does not works. We tried changing other parameters of the job but nothing seems to solve this.

We can also see this in development environment without docker. In development we left the worker, admin and flower running and only restarted celery beat. Same behavior. We need to disable/enable to get this working.

How can we get a consistent behavior with celery-beat?

amitjindal avatar Oct 18 '22 06:10 amitjindal

Don't use this django-celery-beat .default PersistentScheduler Works fine, django-celery-bea package will just make you pull your hairs.

kedarcode avatar Jun 20 '23 09:06 kedarcode

Can confirm it still happens for v2.5.0. Removing django_celery_beat.schedulers:DatabaseScheduler works, but will un-link django_celery_results 's periodic task name (will now be None).

kwongtn avatar Jul 20 '23 06:07 kwongtn

Hi @ all, i'm facing the same issue with my setup which is similar to the initial reported one. Is there any progress on this topic or workaround for this (except not using the DatabaseScheduler)? Thanks & BR

fsupper avatar Aug 09 '23 07:08 fsupper

Meanwhile i have tested this without using docker. celery 5.2.3 django-celery-beat 2.2.1

Get the same result - Task was not be send to celery worker.

$ celery -A project beat --scheduler django_celery_beat.schedulers:DatabaseScheduler -l DEBUG
celery beat v5.2.3 (dawn-chorus) is starting.
__    -    ... __   -        _
LocalTime -> 2023-08-09 12:33:55
Configuration ->
    . broker -> amqp://rabbituser:**@localhost:5672//
    . loader -> celery.loaders.app.AppLoader
    . scheduler -> django_celery_beat.schedulers.DatabaseScheduler

    . logfile -> [stderr]@%DEBUG
    . maxinterval -> 5.00 seconds (5s)
[2023-08-09 12:33:55,476: DEBUG/MainProcess] DatabaseScheduler: initial read
[2023-08-09 12:33:55,476: DEBUG/MainProcess] Writing entries...
[2023-08-09 12:33:55,477: DEBUG/MainProcess] DatabaseScheduler: Fetching database schedule
[2023-08-09 12:33:55,484: DEBUG/MainProcess] Current schedule:
<ModelEntry: celery.backend_cleanup celery.backend_cleanup(*[], **{}) <crontab: 0 4
         * *
          * (m/h/d/dM/MY), Europe/Vienna>
        >
<ModelEntry: Test BEAT in DEV project.apps.administration.tasks.test_celery_beat_in_dev(*[], **{}) <freq: 10.00 seconds>>
[2023-08-09 12:34:00,510: DEBUG/MainProcess] Writing entries...

$ celery -A project worker --scheduler django_celery_beat.schedulers:DatabaseScheduler -l DEBUG
 
 -------------- celery@hostxyz v5.2.3 (dawn-chorus)
--- ***** ----- 
-- ******* ---- Linux-5.15.0-76-generic-x86_64-with-glibc2.31 2023-08-09 12:37:24
- *** --- * --- 
- ** ---------- [config]
- ** ---------- .> app:         project:0x7fd74228d700
- ** ---------- .> transport:   amqp://rabbituser:**@localhost:5672//
- ** ---------- .> results:     
- *** --- * --- .> concurrency: 12 (prefork)
-- ******* ---- .> task events: OFF (enable -E to monitor tasks in this worker)
--- ***** ----- 
 -------------- [queues]
                .> celery           exchange=celery(direct) key=celery

[tasks]
  . celery.accumulate
  . celery.backend_cleanup
  . celery.chain
  . celery.chord
  . celery.chord_unlock
  . celery.chunks
  . celery.group
  . celery.map
  . celery.starmap

[2023-08-09 12:37:25,431: INFO/MainProcess] Connected to amqp://rabbituser:**@127.0.0.1:5672//
[2023-08-09 12:37:25,437: INFO/MainProcess] mingle: searching for neighbors
[2023-08-09 12:37:26,456: INFO/MainProcess] mingle: all alone
[2023-08-09 12:37:26,484: WARNING/MainProcess] /home/userxyz/.pyenv/versions/3.9.6/envs/venv_3.9.6/lib/python3.9/site-packages/celery/fixups/django.py:203: UserWarning: Using settings.DEBUG leads to a memory
            leak, never use this setting in production environments!
  warnings.warn('''Using settings.DEBUG leads to a memory

I have created a Interval schedule which starts every 10 seconds to print out some text. Only when i disable and re enable the task, it proceeds.

fsupper avatar Aug 10 '23 11:08 fsupper

This still happens with

  • celery 5.3.1
  • django-celery-beat 2.5.0

Manual or automatic runs were not happening. What worked for us was changing an interval to a crontab and running it again.

I don't feel confident at all with this though, and would appreciate progress with this issue.

gdvalderrama avatar Aug 31 '23 06:08 gdvalderrama

same here, it's not reliable at all, I keep enabling/disabling until it works any progress ?

rednaks avatar Sep 20 '23 17:09 rednaks

me too

nciefeiniu avatar Sep 21 '23 15:09 nciefeiniu

таже хуйня

edpyt avatar Sep 22 '23 13:09 edpyt

Oh, everything is good. just switched from sqlite db to postgre db and fixed it. 😅

edpyt avatar Sep 27 '23 16:09 edpyt

Oh, everything is good. just switched from sqlite db to postgre db and fixed it. 😅

Not my case, I've been using Postgres from the start 😞

gdvalderrama avatar Sep 28 '23 07:09 gdvalderrama

Is there any recent progress on resolving this issue? Alternatively, does anyone know of a workaround? I'm encountering the same problem and need to enable/disable it to make it work again.

vaaibhavsharma avatar Oct 28 '23 10:10 vaaibhavsharma

I was having the same issue where the tasks weren't being executed by the beat, and the problem has been fixed after changing this:

@app.on_after_configure.connect
def setup_periodic_tasks(sender, **kwargs):
    sender.add_periodic_task(crontab(), test())

by this:

app.conf.beat_schedule = {
    'taks-id': {
        'task': 'full.task.path.test',
        'schedule': crontab(),
    },
}

edit: the celery app is being imported from the app configured on main.celeryapp with the proper configuration:

app = Celery("main")
app.config_from_object("main.settings.celery", namespace="CELERY")
app.autodiscover_tasks()

luabida avatar Nov 14 '23 19:11 luabida