django-celery-beat icon indicating copy to clipboard operation
django-celery-beat copied to clipboard

Published 20 hours ago • verified publisher icon celery

Celery beat scheduler - Multiple Task Executions

Open vpogrebi opened this issue 2 years ago • 22 comments

Checklist

  • [x] I have verified that the issue exists against the master branch of Celery.
  • [x] This has already been asked to the discussions forum first.
  • [x] I have read the relevant section in the contribution guide on reporting bugs.
  • [x] I have checked the issues list for similar or identical bug reports.
  • [x] I have checked the pull requests list for existing proposed fixes.
  • [x] I have checked the commit log to find out if the bug was already fixed in the master branch.
  • [x] I have included all related issues and possible duplicate issues in this issue (If there are none, check this box anyway).

Mandatory Debugging Information

  • [x] I have included the output of celery -A proj report in the issue. (if you are not able to do this, then at least specify the Celery version affected).
  • [x] I have verified that the issue exists against the master branch of Celery.
  • [x] I have included the contents of pip freeze in the issue.
  • [x] I have included all the versions of all the external dependencies required to reproduce this bug.

Optional Debugging Information

  • [ ] I have tried reproducing the issue on more than one Python version and/or implementation.
  • [ ] I have tried reproducing the issue on more than one message broker and/or result backend.
  • [ ] I have tried reproducing the issue on more than one version of the message broker and/or result backend.
  • [ ] I have tried reproducing the issue on more than one operating system.
  • [ ] I have tried reproducing the issue on more than one workers pool.
  • [ ] I have tried reproducing the issue with autoscaling, retries, ETA/Countdown & rate limits disabled.
  • [ ] I have tried reproducing the issue after downgrading and/or upgrading Celery and its dependencies.

Related Issues and Possible Duplicates

Related Issues

  • None

Possible Duplicates

  • None

Environment & Settings

Celery version: 5.2.7

celery report Output: 

software -> celery:5.2.7 (dawn-chorus) kombu:5.2.4 py:3.8.13
            billiard:3.6.4.0 redis:4.3.1
platform -> system:Linux arch:64bit, ELF
            kernel version:3.10.0-1160.62.1.el7.x86_64 imp:CPython
loader   -> celery.loaders.default.Loader
settings -> transport:redis results:disabled

deprecated_settings: None

Steps to Reproduce

  1. Schedule two (or more) tasks to start at the same time
  2. When one task completes, scheduler sync (tick()) is executed
  3. If the other task is still running, is_due() returns True - resulting in tick() execute self.apply_entry() which essentially starts same task (the one that is already executing) to execute again

Required Dependencies

  • Minimal Python Version: 3.8
  • Minimal Celery Version: 5.2.7 (not necessarily minimal)
  • Minimal Kombu Version: 5.2.4
  • Minimal Broker Version: N/A or Unknown
  • Minimal Result Backend Version: N/A or Unknown
  • Minimal OS and/or Kernel Version: Linux
  • Minimal Broker Client Version: N/A or Unknown
  • Minimal Result Backend Client Version: N/A or Unknown

Python Packages

pip freeze Output: 

amqp @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/94/d8/7787846037071cf203833b4891393d7fee750673537eccf83374f4e68a/amqp-5.1.1-py3-none-any.whl
aniso8601 @ file:///data/zkci8im/.cache/pypoetry/artifacts/cb/bb/0b/f3e74131c6390dbb7622617f9c8a590661cd4910e2e342c07ab40fe8d5/aniso8601-7.0.0-py2.py3-none-any.whl
ansible==5.1.0
ansible-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/b4/59/459ce5c2e21433c1443caaecc97cbaa260085761ef58ad0794f9ef1894/ansible-core-2.12.6.tar.gz
asgiref @ file:///data/zkci8im/.cache/pypoetry/artifacts/60/d3/77/8e5b9c433ba4707af49c8a8879a4792689501bc21468d866dcba293387/asgiref-3.5.2-py3-none-any.whl
async-timeout @ file:///data/zkci8im/.cache/pypoetry/artifacts/9e/20/3e/134e90bed6cb6814ec11555be6900e30ab163641792ec5003bd6aa59d7/async_timeout-4.0.2-py3-none-any.whl
attrs @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/b8/3f/509c2f0fdcb7f29b8a253972760d0c7830bb0b0403328ae0eab3cea799/attrs-21.4.0-py2.py3-none-any.whl
billiard @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/9f/c2/e3fc46d8ea60110a6d6c6b23cabe49bb06db28d7ef3fcef9f560c37eb1/billiard-3.6.4.0-py3-none-any.whl
boto3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/2a/5a/b2c1f3b722c8e02330dba16637ed5b9f12d195bef6226e2d72e6112a7b/boto3-1.23.10-py3-none-any.whl
botocore @ file:///data/zkci8im/.cache/pypoetry/artifacts/10/6b/56/b501c514fffa9d9a45ecf0d334e7717c7d963cae46d6cdfd1c90d0a192/botocore-1.26.10-py3-none-any.whl
cached-property @ file:///data/zkci8im/.cache/pypoetry/artifacts/c4/6c/91/92f6a180669978ba5b5cfeea34239598e4a62f798452e667bab474eca3/cached_property-1.5.2-py2.py3-none-any.whl
celery @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/72/4d/52e0f4d27000064a2266f9d939f1a9e6e8a385310ea28ef12102c76768/celery-5.2.7-py3-none-any.whl
certifi @ file:///data/zkci8im/.cache/pypoetry/artifacts/9a/24/91/91ca3aa0f55638b8b2e2e0751f2e1cba1f342c36d4eea2028b6fc96431/certifi-2022.5.18.1-py3-none-any.whl
cffi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/1b/a4/670c457d4f006c93740689d474309dfa24a89159e27073f4b3fc44bb98/cffi-1.15.0-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
charset-normalizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/e6/c2/e485049fad66be34cb361166725f59e4e9ef24804c75713cbb3bd6699d/charset_normalizer-2.0.12-py3-none-any.whl
click @ file:///data/zkci8im/.cache/pypoetry/artifacts/e2/03/35/a5b93ff313b614d10ef8c1dfecb531470309e3799cbf6fbfeed0a1ce0d/click-8.1.3-py3-none-any.whl
click-didyoumean @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/30/2c/da96c86e55be9b67c908b36e64e815e3c9db43db453e65b2f57ef7f52d/click_didyoumean-0.3.0-py3-none-any.whl
click-plugins @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/16/38/b7c58246dc79c9091376c792a8981f86944dd0ebe9581a58ba9a5ce0ea/click_plugins-1.1.1-py2.py3-none-any.whl
click-repl @ file:///data/zkci8im/.cache/pypoetry/artifacts/f2/c0/d5/b58b59f548e7a632f7780eae3f73f0258eb1e926d52f0170c21304ad4b/click_repl-0.2.0-py3-none-any.whl
colorama @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/e1/ed/183662e38ad81210d1ff1a86d8ce34409a7d6a1f7d0ef629715dab9344/colorama-0.4.4-py2.py3-none-any.whl
coreapi @ file:///data/zkci8im/.cache/pypoetry/artifacts/e0/15/0a/3368989abeaa634a32dd92a149db85845315e1528a202d163bd8def96b/coreapi-2.3.3-py2.py3-none-any.whl
coreschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/20/6f/08c281ffb0c8eff16526a9b74d5af61dbcac6172ac2605f4db6c4c480a/coreschema-0.0.4.tar.gz
cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/dd/ef/07/3dd5838cbc8f2b3a22dc4db8399410fe13a6a81f32ce2b40d2c946c788/cryptography-37.0.2-cp36-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
cx-Oracle @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/a1/48/01f42f211f534f73aed19b07156e2f58bc51f3e931b7ff24e916bc6183/cx_Oracle-8.3.0-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
cycler @ file:///data/zkci8im/.cache/pypoetry/artifacts/32/5f/b5/69e6bd574bcff38cf319bbc2937a6f71f47b5532fff18dfd7dc221591a/cycler-0.11.0-py3-none-any.whl
defer @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/d4/64/12aa60c68128ded2c3b83733ede1376668571bee98f2234f998d6fccfd/defer-1.0.4.tar.gz
defusedxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/8e/a8/6b6077e99f00435c289f1e6d72413a518d187c42f4f16018e9748996aa/defusedxml-0.7.1-py2.py3-none-any.whl
Deprecated @ file:///data/zkci8im/.cache/pypoetry/artifacts/1b/df/01/0a69c6fa606522ad7535a9433d6f353e57e674a41f23b0c75851455ade/Deprecated-1.2.13-py2.py3-none-any.whl
diffsync @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/f2/03/71d9fcbe3efd41c93400434afd2eecc791427d325371195d79b0ecba7e/diffsync-1.4.3-py3-none-any.whl
Django @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/b8/62/2618b9e6130ebb2661834718637fd186dcaf9d4bbdbe5d8e40b5044e90/Django-3.1.14-py3-none-any.whl
django-ajax-tables @ file:///data/zkci8im/.cache/pypoetry/artifacts/41/4b/c2/0b226c181d25ebe682f1c331ff835fc913da1840e63970f43b2d4d5025/django_ajax_tables-1.1.1-py3-none-any.whl
django-appconf @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/b0/42/820bd462dcd763df6043e873d416d949595b144ac76bda950a7059322e/django_appconf-1.0.5-py3-none-any.whl
django-cacheops @ file:///data/zkci8im/.cache/pypoetry/artifacts/77/ab/1a/858382fe25b8025eddd02dc56b2ed986f86d9343e51186984d4c87de0b/django_cacheops-5.1-py2.py3-none-any.whl
django-celery-beat @ file:///data/zkci8im/.cache/pypoetry/artifacts/3f/71/03/feac735288ca43683afd8f5e889f6c1670345f879acfb60f5391f4f559/django_celery_beat-2.2.1-py2.py3-none-any.whl
django-constance @ file:///data/zkci8im/.cache/pypoetry/artifacts/92/73/f6/b27afffb80673c8e7cff1f06405d06d8078cce3ccf8156b94349e7417e/django_constance-2.8.0-py3-none-any.whl
django-cors-headers @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/95/72/c2b1d92874644571f8e4b3b90951706ac3d96db0ae5f8ed50a196fc1b1/django_cors_headers-3.7.0-py3-none-any.whl
django-cryptography @ file:///data/zkci8im/.cache/pypoetry/artifacts/be/36/af/6730dd6bebebfddbcac7ffe452b9b6db03977659dad14b31f5417f1f85/django_cryptography-1.0-py3-none-any.whl
django-db-file-storage @ file:///data/zkci8im/.cache/pypoetry/artifacts/73/ed/f1/84bb26eaf28ee7772f62bf319f95bc47b58964c9c45758b73b7df111ee/django-db-file-storage-0.5.5.tar.gz
django-downloadview @ file:///data/zkci8im/.cache/pypoetry/artifacts/b1/2e/86/bbffe2f5c2568e7f427eb695e6b21f312104be6bbd968b66012ff5804a/django_downloadview-2.3.0-py3-none-any.whl
django-entangled @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/37/48/145379ef54ff01cb65af0eb2397a87e83fdf209dcc1aaebb1e4a913bbd/django_entangled-0.5.3-py3-none-any.whl
django-extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/44/36/92d6e2883b730fa9f5870d82725511e42c87732a524094cef75e9080b9/django_extensions-3.1.5-py3-none-any.whl
django-filter @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/56/fc/d46f2e843498b83624f789ec451754d9caa1da8e04dafcbbb61d273a1e/django_filter-2.4.0-py3-none-any.whl
django-health-check @ file:///data/zkci8im/.cache/pypoetry/artifacts/68/fa/24/e2b9ae1477deafa6b5dcbf3878000c6eca08b90da6cbb2ef09b31cfe3a/django_health_check-3.16.5-py2.py3-none-any.whl
django-jinja @ file:///data/zkci8im/.cache/pypoetry/artifacts/7d/7d/e5/ebaf16f467f945ac40af64aa5470aa6c16b6d15d1bba0d941c128ddf3d/django_jinja-2.7.1-py3-none-any.whl
django-js-asset @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/c3/d7/d7ec51ee2330df3de9033568464f80b75a61399948db022bf6a2faf604/django_js_asset-2.0.0-py3-none-any.whl
django-mptt @ file:///data/zkci8im/.cache/pypoetry/artifacts/26/12/ef/a313335a7225e0b4cd0f2b6d6cfe0057523ce7b5fb803295fd2dc7b25a/django_mptt-0.11.0-py2.py3-none-any.whl
django-ordered-model @ file:///data/zkci8im/.cache/pypoetry/artifacts/ac/a1/1c/7deedf732708cd7a331a55900332d66b6377a30637f28de520f7da79d0/django_ordered_model-3.6-py3-none-any.whl
django-picklefield @ file:///data/zkci8im/.cache/pypoetry/artifacts/a9/e2/04/f4ab1ce6c3ba035cb9f3492562ff2772f6bcd522690498e141ea225844/django_picklefield-3.0.1-py3-none-any.whl
django-prometheus @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/08/89/a058c01936e8c1ff21a3c3696258faf6dd5c236ad1ca7ce74ec7a2744a/django_prometheus-2.1.0-py2.py3-none-any.whl
django-redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/e6/f4/01/66c9ff3f089f1e3cd1a4363cf0931618d904c54676ace18cfdf94a3f83/django_redis-5.2.0-py3-none-any.whl
django-rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/ce/18/a5/8ca3fd96dee56aa6a520a9fbd8305feb9fd8cb1eab77d0fff267e2d872/django_rq-2.4.1-py2.py3-none-any.whl
django-storages @ file:///data/zkci8im/.cache/pypoetry/artifacts/84/52/bb/5ac21f52e4e07dcddd223ea08d7ec643a9ae872226ad79b1e4daed3769/django_storages-1.12.3-py3-none-any.whl
django-tables2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/78/46/84/69794e310d82527ae5e95fefc3fdde4d223a76d7e9dcbf58446b61e597/django_tables2-2.3.4-py2.py3-none-any.whl
django-taggit @ file:///data/zkci8im/.cache/pypoetry/artifacts/0a/ba/45/8fd1c16264accb6c7b76eb13fd536bdbc36bce8e5e4f37f9d976a834c0/django_taggit-1.3.0-py3-none-any.whl
django-timezone-field @ file:///data/zkci8im/.cache/pypoetry/artifacts/bc/1a/83/6e3f3fd42c1cb4bc4744298de8f3eee498573bd3a5d45e87b0dd58eb4b/django_timezone_field-4.1.2-py3-none-any.whl
django-webserver @ file:///data/zkci8im/.cache/pypoetry/artifacts/39/e2/f5/581bc4bba43156a12191374c64b3de7e357126ab5a32b51e8c832ad52b/django_webserver-1.2.0-py2.py3-none-any.whl
djangorestframework @ file:///data/zkci8im/.cache/pypoetry/artifacts/a0/f6/64/3c17dd26b71862f114e3edeb3fbbe5a113b97fbabfc02ddc01ac73d325/djangorestframework-3.12.4-py3-none-any.whl
drf-yasg @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/b1/f2/b6d572c8bf1b12ea52c42f019d95c72855926fbf2e4a21158eb68c9425/drf_yasg-1.20.0-py2.py3-none-any.whl
ecdsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/88/fc/5e/0169779073adb3d1920c38c24f1b47601e447622ec221514fdd9cdb1cb/ecdsa-0.17.0-py2.py3-none-any.whl
et-xmlfile @ file:///data/zkci8im/.cache/pypoetry/artifacts/fd/6d/4f/3e758a754df0ad5b0be20ae9e4c3a4d3b07e778b38a0ff76c980cb6f9d/et_xmlfile-1.1.0-py3-none-any.whl
fonttools @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/40/9d/ad9a026cdfc88d8ab195b1228e254f69a91a3a7084d69b4ff79de5ba6d/fonttools-4.33.3-py3-none-any.whl
funcy @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/29/3c/7612d8bae9b3af86333d22fe6156fa13ae2d26c70803da765b7f87f8ed/funcy-1.17-py2.py3-none-any.whl
GES.Entitlements @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/1d/60/7e657bbeee6cc1becde51c9d21ed5618547b8d71a5e42fe2e02c0c2275/GES.Entitlements-7.4.0.0-py3-none-any.whl
gitdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/48/e1/e92559540498e6fc81acc2b79725aed4831869ac4930e4875f042a3bc6/gitdb-4.0.9-py3-none-any.whl
GitPython @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/99/a3/8dc50fe3361368d219d1dc12474ce03ee847dede7e83853f36238f7525/GitPython-3.1.18-py3-none-any.whl
graphene @ file:///data/zkci8im/.cache/pypoetry/artifacts/e4/b4/e5/ebf568635788b1888e94de08f61bf6df7bc035c66fdb8130caccde0534/graphene-2.1.9-py2.py3-none-any.whl
graphene-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/ed/bb/91/d0a9103f49ff61786d14c64d62b62d3bf3ac06aee2f44eed7f19c7d285/graphene_django-2.15.0-py2.py3-none-any.whl
graphene-django-optimizer @ file:///data/zkci8im/.cache/pypoetry/artifacts/e7/41/e6/a86f233609fe6e007737bc3960a4d869720e2d7f6fbbac49fd76a7b90f/graphene-django-optimizer-0.8.0.tar.gz
graphql-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/0a/5d/522bcaafc471de5533ac9aeef689edc9402c048bc5566cc746f7e17928/graphql_core-2.3.2-py2.py3-none-any.whl
graphql-relay @ file:///data/zkci8im/.cache/pypoetry/artifacts/2d/bb/de/db5baddb597438d931a9cb986596b9382ab4153c628523bd52e4446fd6/graphql_relay-2.0.1-py3-none-any.whl
ibm-db @ file:///data/zkci8im/.cache/pypoetry/artifacts/40/f3/30/bf498333533e3962d2dd380fa2c15dd56280b298d4d6d29e57373c2c4e/ibm_db-3.1.1.tar.gz
idna @ file:///data/zkci8im/.cache/pypoetry/artifacts/6f/d1/a9/d0625f343d7d25e3e0effced7539d67c94c0426cd32bdf4fbb3a7528f0/idna-3.3-py3-none-any.whl
importlib-metadata @ file:///data/zkci8im/.cache/pypoetry/artifacts/b6/26/b1/de9225713e00b48ca565477825c09a513cf6e597d5b32a6f11a74f1eb8/importlib_metadata-4.4.0-py3-none-any.whl
inflection @ file:///data/zkci8im/.cache/pypoetry/artifacts/98/bc/19/dd3ca5a6cc7c1d2ebc3fdeb01ae68dc6ec59c3a27f27d6f8ba2ed7560e/inflection-0.5.1-py2.py3-none-any.whl
isodate @ file:///data/zkci8im/.cache/pypoetry/artifacts/7f/4c/a9/6ee9a5be06dbc032cddbdd42142dc7828a67c6d959f9cc7eda6f6cec3a/isodate-0.6.1-py2.py3-none-any.whl
itypes @ file:///data/zkci8im/.cache/pypoetry/artifacts/27/5d/31/a7c551c3439cd7711a903bf67c5313d27a79bb908adcb734413f6ff3b8/itypes-1.2.0-py2.py3-none-any.whl
Jinja2 @ file:///data/zkci8im/.cache/pypoetry/artifacts/50/1b/94/55cae878ce19751af0ec5f40765ef51e5af01c903cad84e0cdeaa3d7f0/Jinja2-2.11.3-py2.py3-none-any.whl
jmespath @ file:///data/zkci8im/.cache/pypoetry/artifacts/65/c5/84/a16f584a42b02584726b43531ed19ca400f538f299a302ba4ad88b4cb3/jmespath-1.0.0-py3-none-any.whl
jsonschema @ file:///data/zkci8im/.cache/pypoetry/artifacts/d7/ea/c1/138bfd39b9e843940605bd7170380281f7e0df8fb2a47fa5d70974fe90/jsonschema-3.2.0-py2.py3-none-any.whl
kiwisolver @ file:///data/zkci8im/.cache/pypoetry/artifacts/90/ec/b4/941f119eada2e8221280a7b23e75a5d3e24ca089327c4cdca627412e85/kiwisolver-1.4.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
kombu @ file:///data/zkci8im/.cache/pypoetry/artifacts/b0/16/df/66aefa2d180ceace98dd7c2b0b1014c4380669e7382214e953aeed23ab/kombu-5.2.4-py3-none-any.whl
lxml @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/b0/67/93757e946c5baba990e7c2c1e16f618c9073f09d6ae033d506db95ce5a/lxml-4.8.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.manylinux_2_24_x86_64.whl
Markdown @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/32/83/707e90466826044af311cf500aca2bc915a5bbecd661201fbce6a7cf77/Markdown-3.3.7-py3-none-any.whl
MarkupSafe @ file:///data/zkci8im/.cache/pypoetry/artifacts/18/2e/dc/4e3a84074e8f7e62f0badbe8fd7e1c027f256893e82d756e7153c758d1/MarkupSafe-2.0.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl
matplotlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/0f/f5/b5/8d4616bd0e92bad21257d238a1cb30c3a643277a1eb7887e274b140ccf/matplotlib-3.5.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
mysqlclient @ file:///data/zkci8im/.cache/pypoetry/artifacts/3e/49/38/62c5c2ca4417ce1aab974d62c8ed023b4731edb2ec0cf30e8700a817d2/mysqlclient-2.1.0.tar.gz
nautobot @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/02/d4/096e1c7aade77e4b2ec96f51bcafd5d384ee2a705847dde9423a24bf8e/nautobot-1.2.11-py3-none-any.whl
nautobot-anap-circuits @ file:///data/zkci8im/.cache/pypoetry/artifacts/63/b5/0d/e8252d47044b6425c000fe541a92a2562ecd22d9a0f24a790fb8b934a4/nautobot_anap_circuits-0.3.15-py3-none-any.whl
nautobot-anap-circuits-sync @ file:///data/zkci8im/.cache/pypoetry/artifacts/4c/c5/58/a610a0576329c19eedcad609c2c4ffd9461819b1e73b80a982eb0ff53c/nautobot_anap_circuits_sync-0.3.6-py3-none-any.whl
nautobot-anap-symphony @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/39/86/f78b31665c81bdddf246d3272a5907d4996a6c861c80f92e9f1dc3455a/nautobot_anap_symphony-0.6.44-py3-none-any.whl
nautobot-bofa-ipam @ file:///data/zkci8im/.cache/pypoetry/artifacts/81/a8/a2/96f2ac7d2200b988b7a731f8e45efef6c3263fb7b823175e9d0e578e29/nautobot_bofa_ipam-1.4.3-py3-none-any.whl
nautobot-bpa @ file:///data/zkci8im/.cache/pypoetry/artifacts/22/d0/80/7de73e8e798b5c9f6a99223f59db1ec60dbaf6da104c56769a014d3e20/nautobot_bpa-2.1.0-py3-none-any.whl
nautobot-cert-letters @ file:///data/zkci8im/.cache/pypoetry/artifacts/ff/a5/0a/2e65e3eebb1f67727efffa57e9616d043e2dd7501d2e432b10fa8ae471/nautobot_cert_letters-0.1.17-py3-none-any.whl
nautobot-clearpass @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/d7/8b/84e2b6ff52cf8a23bee6f935d8cdc881668f43c25afa704d90735894e1/nautobot_clearpass-1.0.9-py3-none-any.whl
nautobot-cmdb @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/90/8a/4e5bbf6bf6b32de2d33e3ba2330ea72cf19e34ee6abaf2fc30a8e4a81d/nautobot_cmdb-1.13.5-py3-none-any.whl
nautobot-common @ file:///data/zkci8im/.cache/pypoetry/artifacts/b5/de/92/7de8b566158ff1df03b33c41f8c87e84eabb8c27fe09d0a94e47e392f3/nautobot_common-0.3.5-py3-none-any.whl
nautobot-cyberark @ file:///data/zkci8im/.cache/pypoetry/artifacts/48/e4/68/148f4623324c647d1733b23fa41b93a906ca17ad4f681d177528773617/nautobot_cyberark-2.2.0-py3-none-any.whl
nautobot-data-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/25/90/36/a34aeabb27539900ee5341cd73aae55f85ee6ac6cb4e43355645965ec2/nautobot_data_classification-1.0.1-py3-none-any.whl
nautobot-data-ownership @ file:///data/zkci8im/.cache/pypoetry/artifacts/52/37/8f/72de92d33a4ff9ac987f52427afa3d7ae7ed222ac34b06885d0c2361c0/nautobot_data_ownership-0.2.0-py3-none-any.whl
nautobot-device-lifecycle-mgmt @ file:///data/zkci8im/.cache/pypoetry/artifacts/d2/c8/5f/cda93c10d2f60155cc1a5fb3e359d291ef2ebf69903032b3524d7cbbbc/nautobot_device_lifecycle_mgmt-0.4.1-py3-none-any.whl
nautobot-eps @ file:///data/zkci8im/.cache/pypoetry/artifacts/8d/86/03/b34e76b9b4c08aca090def035abbb5ec5d56fc30a710d3934c6eb1c2b5/nautobot_eps-0.1.0-py3-none-any.whl
nautobot-network-classification @ file:///data/zkci8im/.cache/pypoetry/artifacts/c8/a6/2a/a00e72efb95bac27c3903d1f04ac888b19383e16c312f447952e67c134/nautobot_network_classification-1.1.0-py3-none-any.whl
nautobot-nidm-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/f8/e1/e03db47852cad4fd24b84c92d6dda32a5d3e68448a66cd23c0eadc5c7d/nautobot_nidm_core-1.2.1-py3-none-any.whl
nautobot-rdr @ file:///data/zkci8im/.cache/pypoetry/artifacts/b9/d0/bb/db1717b5380cc3bb98c70bde3190dd278194e350fc7e1b3ebcbb68492b/nautobot_rdr-1.0.1-py3-none-any.whl
nautobot-sevone-onboarding @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/b0/ee/48d5c3db983b7c208d26b32de54965f275a0408f507dd87294408b4b97/nautobot_sevone_onboarding-0.0.1-py3-none-any.whl
nautobot-ssot @ file:///data/zkci8im/.cache/pypoetry/artifacts/15/19/8a/6aa9f5ad8232fa1c9e196fed6bd113b0d4f7cab6fd67d4a4c50f6a0c94/nautobot_ssot-1.1.2-py3-none-any.whl
nautobot-standards @ file:///data/zkci8im/.cache/pypoetry/artifacts/e3/23/18/61dcab40bb5e3561a0cb412be3917595c317131a9211f19c575cf4ed24/nautobot_standards-1.8.5-py3-none-any.whl
netaddr @ file:///data/zkci8im/.cache/pypoetry/artifacts/f9/16/e8/d9224c3975d478de3009d1d47712cf111b0a4b5d854dc9e9fabc65dbde/netaddr-0.8.0-py2.py3-none-any.whl
netutils @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/db/16/4111ea796f3afb67d33e86f63f35aba5a21019cda3f95e903ba63e8762/netutils-1.0.0-py3-none-any.whl
numpy @ file:///data/zkci8im/.cache/pypoetry/artifacts/de/d5/28/b0cfd2d4c3e9a1c8ae4a78490049d8a3412d697a7f0a5b34c44f4d1d2b/numpy-1.22.4-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/07/0e/03/4277e4ccf4acbe4361be783f58dd746cb95b78649a558edf1307a5a1f6/oauthlib-3.2.0-py3-none-any.whl
openpyxl @ file:///data/zkci8im/.cache/pypoetry/artifacts/e8/b7/32/e56fe150a5abcd4739833b3a0a1548d230679b29074ef42402272629c9/openpyxl-3.0.10-py2.py3-none-any.whl
packaging @ file:///data/zkci8im/.cache/pypoetry/artifacts/3b/2b/d0/229eacac699b7990b07c4a20d0f449f10465f21747f61e1b16ca0788b6/packaging-21.3-py3-none-any.whl
Pillow @ file:///data/zkci8im/.cache/pypoetry/artifacts/47/f1/78/5bbff537c3d88fb8237793032585a247723b14e00240af6cb68de37365/Pillow-9.0.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
platformdirs @ file:///data/zkci8im/.cache/pypoetry/artifacts/45/0a/73/b60f9509a6bc1223523fb621fa706e105b8b75953ba5f6b20671b297ce/platformdirs-2.5.2-py3-none-any.whl
prometheus-client @ file:///data/zkci8im/.cache/pypoetry/artifacts/29/46/d1/ae1047fbfd3702e1bcdc496b5ac6058f6522256abda4f7da8893f32891/prometheus_client-0.14.1-py3-none-any.whl
promise @ file:///data/zkci8im/.cache/pypoetry/artifacts/62/2a/40/f263aa397064cb03a099f84abc98a7f75bdc1a34962f4eb0012ea78a53/promise-2.3.tar.gz
prompt-toolkit @ file:///data/zkci8im/.cache/pypoetry/artifacts/a6/f6/75/fb1c4bae58f6c4622619dc9dbb1733b76f7ff3c0578d91c15ae91feae7/prompt_toolkit-3.0.29-py3-none-any.whl
psycopg2-binary @ file:///data/zkci8im/.cache/pypoetry/artifacts/96/8d/16/3d21fd6c74323042ba23e37cbdb4d0d2a2bcfaa8e50d5e394378ca4a38/psycopg2_binary-2.8.6-cp38-cp38-manylinux1_x86_64.whl
pyasn1 @ file:///data/zkci8im/.cache/pypoetry/artifacts/44/6a/a9/c21cf90c1e0756f209e70aa7f38486615fed1dd0d0aea32e06f20ddb4c/pyasn1-0.4.8-py2.py3-none-any.whl
pycountry @ file:///data/zkci8im/.cache/pypoetry/artifacts/42/ef/f9/4f6319d77b9ae31577947642412faf82f7bfe8bc6a9bd52ea490088cba/pycountry-20.7.3.tar.gz
pycparser @ file:///data/zkci8im/.cache/pypoetry/artifacts/33/0c/a0/326b0a2cf7e136e67ea4dbdbcfe7147a7db9c8eac3f45eaf5120b95303/pycparser-2.21-py2.py3-none-any.whl
pycryptodome @ file:///data/zkci8im/.cache/pypoetry/artifacts/5a/a7/db/1f6483e9468f56bf8129100ace0c108433c24ebb8302620d1db5caa52c/pycryptodome-3.10.4-cp35-abi3-manylinux2010_x86_64.whl
pydantic @ file:///data/zkci8im/.cache/pypoetry/artifacts/6b/fd/fb/a8b33721ecebb58ba4b2fde8b2e799fb03ef090283041c8ff8136cb96c/pydantic-1.9.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
PyJWT @ file:///data/zkci8im/.cache/pypoetry/artifacts/4e/58/c4/35a0ce0b263912f8cb70102a00ea6ad3e4df070fc1999acd502193b950/PyJWT-2.4.0-py3-none-any.whl
pymssql @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/f4/f0/5a68c58d9ad2ca40ecbedd7675d11f7ecc10d2efebb0f7ad38b2d52c4f/pymssql-2.2.5-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
pyparsing @ file:///data/zkci8im/.cache/pypoetry/artifacts/fc/60/07/1b440c7539c1662fbae8276892b463f7ff5daa3dbde20a91709721264d/pyparsing-3.0.9-py3-none-any.whl
pyrsistent @ file:///data/zkci8im/.cache/pypoetry/artifacts/b3/07/67/b2f8ed0cbef09958ed5340a50cfb1649662be07ba14110e999b6cfda0c/pyrsistent-0.18.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
python-crontab @ file:///data/zkci8im/.cache/pypoetry/artifacts/80/4f/88/386611efe39c69c86592f1eb8d3c44f6dc730192df536ca8351a5e7721/python-crontab-2.6.0.tar.gz
python-dateutil @ file:///data/zkci8im/.cache/pypoetry/artifacts/a8/e2/44/bdcaa1ff9495f3ad9e18dd6e63bb9e887647aba953d9ad9fc05a86fae7/python_dateutil-2.8.2-py2.py3-none-any.whl
python-jose @ file:///data/zkci8im/.cache/pypoetry/artifacts/f6/2a/b8/b6494aa866f7cc75ab568808374a5b083ffdc16206e8f3b289ee18c2b0/python_jose-3.3.0-py2.py3-none-any.whl
python3-openid @ file:///data/zkci8im/.cache/pypoetry/artifacts/0c/12/9f/ea7f74cf5238a45aadf81b3b7eadcfefa8951666c12b69fec11aa464bd/python3_openid-3.2.0-py3-none-any.whl
python3-saml @ file:///data/zkci8im/.cache/pypoetry/artifacts/ad/dc/e4/ebc0a921e5a1728077b6deb835157f90121645d5052b614e3982f3d670/python3_saml-1.12.0-py3-none-any.whl
pytz @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/34/d6/c45c9e208f555808c5cd019789091f10dd40a2f3022649a3f20800d62a/pytz-2022.1-py2.py3-none-any.whl
pyuwsgi @ file:///data/zkci8im/.cache/pypoetry/artifacts/94/5a/2a/3ebdbc0013fd6046b29a1b273279cdbe5092fbf951c00a490d69f83c42/pyuwsgi-2.0.20-cp38-cp38-manylinux_2_12_x86_64.manylinux2010_x86_64.whl
PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/3b/da/8e567f27818a41bf1549e53b5cd23647b1d7e2e647e0822dabb01a63bb/PyYAML-5.4.1-cp38-cp38-manylinux1_x86_64.whl
redis @ file:///data/zkci8im/.cache/pypoetry/artifacts/af/f2/d3/bd5e4c2a389a010eee4acc4f08add48552e1be122bc7c768db5b8375ab/redis-4.3.1-py3-none-any.whl
requests @ file:///data/zkci8im/.cache/pypoetry/artifacts/02/17/db/7a356bc4171d71bd271a00d6df46c824f4370f2ef39d88b74e6083bc03/requests-2.27.1-py2.py3-none-any.whl
requests-file @ file:///data/zkci8im/.cache/pypoetry/artifacts/82/c0/ba/0889eaaa9e09451feea27723a96f52056ef402da06ae9e2b0b785d1879/requests_file-1.5.1-py2.py3-none-any.whl
requests-oauthlib @ file:///data/zkci8im/.cache/pypoetry/artifacts/b7/88/21/a294759fa4bda84604868791a9fb7ae94d995a0a5a3e446867437b62ef/requests_oauthlib-1.3.1-py2.py3-none-any.whl
requests-toolbelt @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/d0/4e/12d5b1fabbdf8cc813b65a85b4a44885ddccf80ddbe42cdab87df3c9c4/requests_toolbelt-0.9.1-py2.py3-none-any.whl
resolvelib @ file:///data/zkci8im/.cache/pypoetry/artifacts/95/f0/cd/33d25064e3e20dd4c1ad2fa315e04cff25fd0c6d6ad79b9068a3c33af2/resolvelib-0.5.5-py2.py3-none-any.whl
robot-bpa-sdk @ file:///data/zkci8im/.cache/pypoetry/artifacts/99/87/39/0a25e697d3f52ae3c0b2520d8249d5abb5573d30222ec78f51518d79e4/robot_bpa_sdk-2021.7.15-py3-none-any.whl
rq @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/8c/a8/a5173f44dfe1a74a0593828ff56e0245df984f348daadd3560c5c1024c/rq-1.10.1-py2.py3-none-any.whl
rsa @ file:///data/zkci8im/.cache/pypoetry/artifacts/70/83/7d/515ad00d6559117fd24ee2dd69dbfdaa556c8986e6e35879cabad091d2/rsa-4.8-py3-none-any.whl
ruamel.yaml @ file:///data/zkci8im/.cache/pypoetry/artifacts/d6/a3/d5/50b355606c7e8cbf1d3e88a8da1b29c1957319ceb41a587c45aca1ee5c/ruamel.yaml-0.17.21-py3-none-any.whl
ruamel.yaml.clib @ file:///data/zkci8im/.cache/pypoetry/artifacts/da/9f/c3/0b970a7488666902b22784c2305a321e4fc5176d96eee119d8ea9c053f/ruamel.yaml.clib-0.2.6-cp38-cp38-manylinux1_x86_64.whl
Rx @ file:///data/zkci8im/.cache/pypoetry/artifacts/34/ec/6f/bc1186029e7ad679cf82dde5131f5790bc62d83b3ba2aa0500222de304/Rx-1.6.1-py2.py3-none-any.whl
s3transfer @ file:///data/zkci8im/.cache/pypoetry/artifacts/20/db/c0/28a2baf88f972584bd05f1495d1a7a4c4e0a318b9831c56534a13df0b5/s3transfer-0.5.2-py3-none-any.whl
setuptools-scm @ file:///data/zkci8im/.cache/pypoetry/artifacts/c1/0e/d6/bf340b99a2ade5d92443240719f8a443e7bd88b591bcef74956a16899a/setuptools_scm-6.4.2-py3-none-any.whl
singledispatch @ file:///data/zkci8im/.cache/pypoetry/artifacts/49/64/ca/e60267abe7b859719ae4e08a0422e8f56ad4a98b8767fb76a8272deb99/singledispatch-3.7.0-py2.py3-none-any.whl
six @ file:///data/zkci8im/.cache/pypoetry/artifacts/f7/55/4d/44b471ed2a50fcce09e4bda1a06c8a7ec642c53b5166ce5c0097275802/six-1.16.0-py2.py3-none-any.whl
smmap @ file:///data/zkci8im/.cache/pypoetry/artifacts/57/bf/ea/1042000312501eb13606330aaa12de6e8c3a57ed7d33f1d659eef2e720/smmap-5.0.0-py3-none-any.whl
social-auth-app-django @ file:///data/zkci8im/.cache/pypoetry/artifacts/66/15/43/2212ab82e7ada52755f3131966a17d91a60bdd662aab49f5b1beb5b559/social_auth_app_django-4.0.0-py3-none-any.whl
social-auth-core @ file:///data/zkci8im/.cache/pypoetry/artifacts/c7/fe/5b/dd39fab35925f9c3c27852572b608f78b47335bd33238cd16a47a12110/social_auth_core-4.1.0-py3-none-any.whl
sqlparse @ file:///data/zkci8im/.cache/pypoetry/artifacts/35/2a/c7/b94bd2500225f7c258ebddb92714290de3566d4a03a5f849b4c2b47c7c/sqlparse-0.4.2-py3-none-any.whl
structlog @ file:///data/zkci8im/.cache/pypoetry/artifacts/1d/1c/fd/7df0160dbd1830fdb46c740b0a81ea252fb6a73b0677353ebaa3f0d674/structlog-20.2.0-py2.py3-none-any.whl
svgwrite @ file:///data/zkci8im/.cache/pypoetry/artifacts/2e/b0/bb/551a83c1df6d04ead4a3e7370f68d0b57e02fad656042cad5df2a310ed/svgwrite-1.4.2-py3-none-any.whl
swagger-spec-validator @ file:///data/zkci8im/.cache/pypoetry/artifacts/71/71/63/7a52697831713b439971a032fb83e31dd1df44aa638e90be82a5ac3d5d/swagger_spec_validator-2.7.4-py2.py3-none-any.whl
text-unidecode @ file:///data/zkci8im/.cache/pypoetry/artifacts/38/cc/09/814f7784e36c8140d07513c0e1709fd9a3807cdd27e6be6efbbdb628fa/text_unidecode-1.3-py2.py3-none-any.whl
tomli @ file:///data/zkci8im/.cache/pypoetry/artifacts/aa/1c/82/1fc4d637d7cad4a6d39e21ab22bd7957ec3228f36c499c4b2d6ff33b4e/tomli-2.0.1-py3-none-any.whl
types-PyYAML @ file:///data/zkci8im/.cache/pypoetry/artifacts/85/94/c2/81254c1bdbfd106a966adca8c00f5cb3a20cdcee1cd50975f3fa345019/types_PyYAML-6.0.7-py3-none-any.whl
typing_extensions @ file:///data/zkci8im/.cache/pypoetry/artifacts/0d/6f/04/b8bd0c34c092d479e1f9ee7ac241d6f264cc01db2832e08fd4a8c1fb3e/typing_extensions-4.2.0-py3-none-any.whl
uritemplate @ file:///data/zkci8im/.cache/pypoetry/artifacts/76/08/ec/048fc3fa6c2480f858b637ddfe52729642b79c01e8f47ce663d74df8d7/uritemplate-4.1.1-py2.py3-none-any.whl
urllib3 @ file:///data/zkci8im/.cache/pypoetry/artifacts/69/80/ba/5408bb5fec831682384684c11eab54da6b8dd997fb85871eba696966b2/urllib3-1.26.9-py2.py3-none-any.whl
vine @ file:///data/zkci8im/.cache/pypoetry/artifacts/d0/af/cd/40d4b255297e753565b27ec2cecb352311be0f65fb821c71a02d9ed109/vine-5.0.0-py2.py3-none-any.whl
wcwidth @ file:///data/zkci8im/.cache/pypoetry/artifacts/16/12/ba/6211db7a12b3c00978b51a85563bdb0d710e9397027fa8f9c8025daa76/wcwidth-0.2.5-py2.py3-none-any.whl
wrapt @ file:///data/zkci8im/.cache/pypoetry/artifacts/9b/83/b3/ee7bc92d3d7376c36e018277e1cd90f052e70ce5d76bf5c672ec271d34/wrapt-1.14.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl
xmlsec @ file:///data/zkci8im/.cache/pypoetry/artifacts/d4/76/2c/871220815f373e94acaa810926b0e857a351d3b24e87a258dc72a2e0c1/xmlsec-1.3.12.tar.gz
zeep @ file:///data/zkci8im/.cache/pypoetry/artifacts/3a/55/89/f82f1d16b4e53c6a318a21628bb26e6921baf442bd1cf65eeeb3fe6920/zeep-4.1.0-py2.py3-none-any.whl
zipp @ file:///data/zkci8im/.cache/pypoetry/artifacts/6d/cc/4e/46283fca4ce6eff04a6e6bfe21e67d2c54e2e6bc27ba1d60c7be3bc0da/zipp-3.8.0-py3-none-any.whl

Other Dependencies

Python: 3.8 celery: 5.2.7 django_celery_beat: 2.2.1

Minimally Reproducible Test Case

Expected Behavior

A single task (both one-off, and periodic) - should execute ONCE AND ONLY ONCE .

Actual Behavior

When multiple tasks are started at roughly the same time, and one task completes while other task(s) still running - running task(s) get started again resulting in MULTIPLE TASK EXECUTIONS.

Package Versions

celery: 5.2.7 django_celery_beat: 2.2.1

Issue Description

Celery beat Scheduler (re)starts same task while task is running, resulting in multiple executions of the same task.

Details

Consider celery.beat.Scheduler.tick() implementation (code below is from v5.2.7):

# pylint disable=redefined-outer-name
def tick(self, event_t=event_t, min=min, heappop=heapq.heappop,
             heappush=heapq.heappush):
        """Run a tick - one iteration of the scheduler.

        Executes one due task per call.

        Returns:
            float: preferred delay in seconds for next call.
        """
        adjust = self.adjust
        max_interval = self.max_interval

        if (self._heap is None or
                not self.schedules_equal(self.old_schedulers, self.schedule)):
            self.old_schedulers = copy.copy(self.schedule)
            self.populate_heap()

        H = self._heap

        if not H:
            return max_interval

        event = H[0]
        entry = event[2]
        is_due, next_time_to_run = self.is_due(entry)
        if is_due:
            verify = heappop(H)
            if verify is event:
                next_entry = self.reserve(entry)
                self.apply_entry(entry, producer=self.producer)
                heappush(H, event_t(self._when(next_entry, next_time_to_run),
                                    event[1], next_entry))
                return 0
            else:
                heappush(H, verify)
                return min(verify[0], max_interval)
        return min(adjust(next_time_to_run) or max_interval, max_interval)

In this implementation, self.is_due() is executed and then (if is_due == True is returned) - self.apply_entry() sends event to start task execution to the celery worker (assuming 'verify' is event):

is_due, next_time_to_run = self.is_due(entry)

Now, consider case where given task is already executing... Since task is executing, it is already "due" - therefore, self.is_due() returns True (is_due == True). Scheduler.tick() happily proceeds and executes self.apply_entry() - which essentially starts same task's execution (by the worker) again. This continues on every Scheduler.tick() execution - until self.is_due() returns False - resulting in same task (single scheduled instance of the task) executed multiple times.

Problem summary

Celery beat Scheduler does not "recognize" that given task is already executing (running) - thus, sends event to start same task over and over again.

Additional details

I work on a project where multiple one-off tasks can be executed at any given moment. We have noticed (and proved that to be true) that same task is often executed multiple times (while some - only once). Having same task executed more than once - "kills" this project; it causes many negative side effects that cannot be handled or predicted (at least, not all of them).

We spent a lot of time troubleshooting this issue - which included debugging (use of breakpoints and stepping through) underlying Celery implementation. This troubleshooting led us to understanding the root cause of the issue: Celery does not recognize that task is "running" and solely relies on is_due() response to send event to start task execution.

We have also proved that same case applies not only to one-off tasks, but also to periodic scheduled tasks: periodic task that should run let's say once a day, or once an hour - can get executed multiple times during same day (or same hour) while initial task is running.

Conclusion

This is a HUGE issue (at least for us). Single task (single "instance" of a task) - must be executed ONCE AND ONLY ONCE. Celery scheduler's inability to recognize that task is already executing and prevent subsequent executions - is a VERY SUBSTANTIAL BUG, causing lots of unpredictable side effects.

To resolve this issue, some sort of "task is running" recognition must be implemented (either within Scheduler.tick() or (more likely) within is_due() method - that recognizes that task is already running and prevents subsequent task executions (while it is actually "is due").

NOTICE: Django celery beat is used, so particular is_due() method that gets executed - is: django_celery_beat.schedulers.ModelEntry.is_due().

Steps Taken

Following steps have been taken while trying to eliminate this issue:

  • CELERY_ENABLE_UTC config. parameter set to True - to eliminate possible TZ related schedule discrepancies (this suggestion is reported in many discussion threads)
  • --task-events command line argument is added to 'celery worker' command - to ensure that worker reports task events
  • An attempt was made to set CELERYBEAT_SYNC_EVERY to 1 - to ensure tick() is executed after each task's completion

Steps to reproduce

  1. Schedule two (or more) tasks to start at roughly the same time
  2. When one task completes, scheduler sync (tick()) is executed
  3. If the other task is still running, is_due() returns True - resulting in tick() execute self.apply_entry() which essentially starts same task (the one that is already executing) to execute again

vpogrebi avatar Jun 16 '22 19:06 vpogrebi

Hey @vpogrebi :wave:, Thank you for opening an issue. We will get back to you as soon as we can. Also, check out our Open Collective and consider backing us - every little helps!

We also offer priority support for our sponsors. If you require immediate assistance please consider sponsoring us.

open-collective-bot[bot] avatar Jun 16 '22 19:06 open-collective-bot[bot]

Sample celery beat log (portion) for demonstration:

[2022-06-07 17:04:11,297: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:11,602: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-enable-2022-06-07-17:04:09.182051_b70fd6d8-4f3a-4d8d-8069-0b0d09ddc4f9 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:11,785: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)
[2022-06-07 17:04:17,094: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2022-06-07 17:04:17,490: INFO/MainProcess] Scheduler: Sending due task AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57 (nautobot.extras.jobs.scheduled_job_handler)

In this log - observe following:

  1. Two tasks were started at the same time ("Scheduler: Sending due task" logs at 2022-06-07 17:04:11)
  2. Few seconds later, one task completed - indicated by the "DatabaseScheduler: Schedule changed" log (2022-06-07 17:04:17,094)
  3. This schedule change event triggered scheduler sync - which in turn executed tick() call with event argument corresponding to the second (still running) task.
  4. In tick(), self.is_due(entry) returned True (task is due) - which resulted in scheduler send another message to start this second task: "Scheduler: Sending due task" log at 2022-06-07 17:04:17,490

End result - one of the tasks (AZMSABG01R1-console-2022-06-07-17:04:07.342526_d2e2aa52-db47-475c-b934-c12d43d91c57) executed twice.

Additional thoughts

I noticed that every time this happens - two (or more) tasks are being kicked off by the same (single) sync event:

  • Single "DatabaseScheduler: Schedule changed" log followed by ...
  • Multiple (two or more) "Scheduler: Sending due task" logs

I am not sure if this fact has anything to do with this issue (bug) - but somehow I have strong feeling that it does. I suspect that these multiple tasks get "associated" with the same (single) event, and when one of the tasks completes - sync (tick()) evaluates each of the other tasks (schedule entries), discovers that they are "due" (is_due() returns True) and sends message to (re)start them again.

I may be wrong, but... this is one suspicion that I've got after giving this issue some more thought.

vpogrebi avatar Jun 16 '22 20:06 vpogrebi

you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?

auvipy avatar Jun 17 '22 07:06 auvipy

@auvipy

you could just modify the previous issue. btw can you please try & check new release https://github.com/celery/django-celery-beat/releases/tag/v2.3.0 ?

Turns out someone else from the team has upgraded our dev lane to django_celery_beat v2.3.0 - and that did not help.

vpogrebi avatar Jun 17 '22 15:06 vpogrebi

One other thought... django_celery_beat.schedulers.ModelEntry.is_due() - has this logic:

        # ONE OFF TASK: Disable one off tasks after they've ran once
        if self.model.one_off and self.model.enabled \
                and self.model.total_run_count > 0:
            self.model.enabled = False
            self.model.total_run_count = 0  # Reset
            self.model.no_changes = False  # Mark the model entry as changed
            self.model.save()
            # Don't recheck
            return schedules.schedstate(False, NEVER_CHECK_TIMEOUT)

When testing, I noticed that self.total_run_count does not match self.model.total_run_count when task (that is already running) is evaluated again - which results in starting same task again. I was able to test my suspicion by providing a custom override of celery.beat.Scheduler.apply_async() that offsets (corrects) this situation:

    def apply_async(self, entry, producer=None, advance=True, **kwargs):
        """Send event to the worker to start task execution.
       
        This is an override of the celery.beat.Scheduler.apply_async() method.
        After executing original apply_async() call, it synchronizes 'total_run_count'
        and saves model. This prevents same task from being started again while it is running.
        """
        resp = super().apply_async(entry, producer=None, advance=True, **kwargs)
        if entry.total_run_count != entry.model.total_run_count:
            entry.total_run_count = entry.model.total_run_count
            entry.model.save()
        return resp

Using this custom override - resolves this issue!

I was able to prove that by ensuring that self.total_run_count and self.model.total_run_count are same at a time apply_async() is executed - somehow prevents multiple executions of the same task. Therefore, I suspect that ensuring that these two versions of total_run_count match at a time apply_async() is executed - somehow ensures that self.model.total_run_count > 0 within django_celery_beat.schedulers.ModelEntry.is_due() which prevents duplicate task execution.

vpogrebi avatar Jun 21 '22 19:06 vpogrebi

I think we can move this issue to django-celery-beat now :)

auvipy avatar Jun 22 '22 07:06 auvipy

There is one other issue that is being resolved by the above "override" (listed in a comment above)... Before (the override), total_run_count stored in the database after job's execution - was unreliable: for the job that ran and completed - total_run_count could be 0, 1, 2, etc. For the job that executed once - it was either 0 or 1, and I could not figure out the "pattern", or predict in advance (while the job was running) whether it would end up with 0 or 1.

Somehow, my override also fixes that issue, too - all total_run_count are 1 for the one-off jobs.

vpogrebi avatar Jun 22 '22 13:06 vpogrebi

Are there any plans to fix these issues in any reasonable timeframe? Months or less? As far as I can tell, celery, especially celery beat, is fundamentally broken. It is not even able to know if a task is already running or not, and this causes issue, upon issue, upon issue, down the road.

There have been comments about this going back half a decade, with no root solution in sight. These issues and core fundamentals should have been thoughtfully and robustly tested with units-testing from day one with sound software engineering.

And please don't say if you have an issue; fork it or put in a pull request. At a certain point, the problems are too numerous and deep to fix. That's why I am asking if there is a plan to resolve these issues soon, or should we cut our losses and move on?

ja-gooding avatar Sep 15 '22 21:09 ja-gooding

I just upgraded to django-celery-beat==2.4.0 with celery==5.2.3, and that didn't help. now tasks duplicate even more.

Does anybody know how to solve this issue?

pythobot avatar Oct 27 '22 07:10 pythobot

Hi Team,

Is it still unresolved ? I am still facing the above stated issue.

I think the hotfix has been applied to apply_async, delay still has the same issue. apply_async seems to work fine with this fix added.

mishra-ankit-dev avatar Jan 20 '23 21:01 mishra-ankit-dev

Hello Team,

Is this still not resolved?

Maryam077 avatar Jun 12 '23 06:06 Maryam077

Try it.

https://github.com/celery/django-celery-beat/pull/660

BaiJiangJie avatar Jul 11 '23 02:07 BaiJiangJie

Try it.

#660

i would like your feedback on this

auvipy avatar Jul 25 '23 08:07 auvipy

@auvipy is there a sure way to reproduce this issue. The way it's described in the original post doesn't always lead to multiple trigger of the same task. I do have the same issue in production but can't identify what is causing the issue. It appears randomly. I would be happy to take the new release for a spin but need a baseline for testing.

guillaumeldc avatar Aug 15 '23 19:08 guillaumeldc

Ok, I finally found a way to consistently reproduce the error. I will do some more testing but wanted to provide the code I'm using to reproduce it. What @vpogrebi is mentioning about having the duplicates around the "schedule changed" is correct and this is what is causing the issue. Here are the logs of the code executions, as you can see task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 is sent twice at the beginning and at the end.

[2023-08-15 23:17:00,071: INFO/MainProcess] Scheduler: Sending due task task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,080: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,085: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,121: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,142: INFO/MainProcess] Scheduler: Sending due task task_3_2_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,159: INFO/MainProcess] Scheduler: Sending due task task_3_3_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,163: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,192: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,214: INFO/MainProcess] Scheduler: Sending due task task_3_4_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,220: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,301: INFO/MainProcess] Scheduler: Sending due task task_3_5_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,308: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,340: INFO/MainProcess] Scheduler: Sending due task task_3_6_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,346: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,379: INFO/MainProcess] Scheduler: Sending due task task_3_7_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,384: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,416: INFO/MainProcess] Scheduler: Sending due task task_3_8_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,422: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,450: INFO/MainProcess] Scheduler: Sending due task task_3_9_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,456: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,481: INFO/MainProcess] Scheduler: Sending due task task_3_10_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,486: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,511: INFO/MainProcess] Scheduler: Sending due task task_3_11_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,516: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,538: INFO/MainProcess] Scheduler: Sending due task task_3_12_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,542: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,568: INFO/MainProcess] Scheduler: Sending due task task_3_13_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,571: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,594: INFO/MainProcess] Scheduler: Sending due task task_3_14_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,598: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,623: INFO/MainProcess] Scheduler: Sending due task task_3_15_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,628: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,649: INFO/MainProcess] Scheduler: Sending due task task_3_16_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,655: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,677: INFO/MainProcess] Scheduler: Sending due task task_3_17_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,681: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,701: INFO/MainProcess] Scheduler: Sending due task task_3_18_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,705: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,729: INFO/MainProcess] Scheduler: Sending due task task_3_19_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,733: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,758: INFO/MainProcess] Scheduler: Sending due task task_3_20_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)
[2023-08-15 23:17:00,763: INFO/MainProcess] DatabaseScheduler: Schedule changed.
[2023-08-15 23:17:00,784: INFO/MainProcess] Scheduler: Sending due task task_3_1_da8aa785-81a8-4d15-b4f0-d0a37cc17e32 (send_task)

You can use this code to reproduce:

def sleep_until(target_datetime):
    """
    Pauses the script until the specified datetime is reached.

    :param target_datetime: datetime.datetime object specifying the time to wait until
    """
    now = timezone.now()
    duration = (target_datetime - now).total_seconds()
    if duration > 0:  # Only sleep if the target datetime is in the future
        time.sleep(duration)


def django_celery_beat_test(nb_tasks=5, nb_batches=5):
    now = timezone.now().replace(second=0, microsecond=0)
    iter_id = uuid.uuid4()
    for i in range(1, nb_batches + 1):
        ps = []
        clocked_time = now + timedelta(minutes=i)
        clocked_schedule = ClockedSchedule.objects.get_or_create(
            clocked_time=clocked_time,
        )[0]
        for x in range(1, nb_tasks + 1):
            task_name = f"task_{i}_{x}_{iter_id}"
            ps.append(
                PeriodicTask(
                    name=task_name,
                    enabled=True,
                    clocked=clocked_schedule,
                    one_off=True,
                    task="send_task",
                    expires=clocked_time + timedelta(minutes=5),
                    kwargs=json.dumps(
                        {
                            "source_model": "stream",
                        }
                    ),
                )
            )
        PeriodicTask.objects.bulk_create(ps)
        for task in ps:
            PeriodicTasks.changed(task)
        sleep_until(clocked_time)

send_task is as follow:

@app.task(name="send_task", bind=True, base=AbortableTask, track_started=True)
def send_task(self, source_mode=None):
    import time
    import random
    print(f"id is: {self.request.id}")
    time.sleep(random.randint(10, 90))
    pass

guillaumeldc avatar Aug 15 '23 23:08 guillaumeldc

@auvipy I tested the main branch briefly using the method I'm highlighting above.

  • On 3 iterations, the current version is consistently duplicating at least one task (always the first one fired).
  • On 3 iterations, using the same parameters (nb_tasks and nb_batch), the code for the main branch is NOT duplicating any task!

It's just 3 iterations, but I can run this all night long and analyze the logs if needed. To parse the logs, you can use this.

def find_duplicate_sequences(text, sequence_length, starting_pattern=""):
    """
    Find duplicated sequences in a long text that begin with a certain pattern.

    :param text: Text to search for duplicated sequences
    :param sequence_length: Length of sequences to search for (including starting pattern)
    :param starting_pattern: Initial pattern to start the sequence with
    :return: Set of duplicated sequences
    """
    if len(starting_pattern) >= sequence_length:
        raise ValueError(
            "The starting pattern is equal or longer than the sequence length!"
        )

    seen_sequences = set()
    duplicated_sequences = set()

    # Adjust sequence length based on the starting pattern's length
    sequence_length -= len(starting_pattern)

    for i in range(0, len(text) - sequence_length + 1):
        if text[i : i + len(starting_pattern)] == starting_pattern:
            sequence = text[i : i + sequence_length + len(starting_pattern)]
            if sequence in seen_sequences:
                duplicated_sequences.add(sequence)
            seen_sequences.add(sequence)

    return duplicated_sequences

guillaumeldc avatar Aug 16 '23 00:08 guillaumeldc

so is it apparent that a new release could some how mitigate this issue?

auvipy avatar Aug 16 '23 05:08 auvipy

Well based on my initial findings, yes. Now keep in mind that we need to see results on a long term analysis which I would be happy to report on when deployed at scale.

guillaumeldc avatar Aug 16 '23 10:08 guillaumeldc

thank you for your kind feedback

auvipy avatar Aug 17 '23 03:08 auvipy

Hi! Do you plan to release a new version anytime soon ?

sar5430 avatar Nov 08 '23 14:11 sar5430

Hi any news on this? I have the same problem and sometimes a task that imports stuff, runs twice, importing the double of the data!

AlessandroSalvetti avatar Mar 12 '24 11:03 AlessandroSalvetti

According to the release notes of the 2.6.0 version on March 3, the PR https://github.com/celery/django-celery-beat/pull/660 that solves https://github.com/celery/django-celery-beat/issues/388 and according to the comments here, also this one. @guillaumeldc Do you still have this issue?

alexgmin avatar Apr 25 '24 13:04 alexgmin