Chords not supported when using 'django-db' backend

[ShuangLau]

Similar to the issue mentioned here: https://github.com/celery/django-celery-results/issues/52#issuecomment-608443564. We have met the same error when trying to run chord tasks using django-db as the result backend:

NotImplementedError
Starting chords requires a result backend to be configured. Note that a group chained with a task is also upgraded to be a chord, as this pattern requires synchronization. Result backends that supports chords: Redis, Database, Memcached, and mor

Here is the lib versions:

Django==3.2.13
django-celery-results==2.3.1
django-health-check==3.16.5
django-celery-beat==2.3.0
psycopg2-binary==2.9.1
django-extensions==3.1.5
gunicorn==20.1.0
django-cors-headers==3.13.0
celery==5.2.3

The setting we configured:

CELERY_BROKER_URL = f'redis://{os.getenv("REDIS_HOST")}:{os.getenv("REDIS_PORT")}'
CELERY_RESULT_BACKEND = "django-db"
CELERY_CACHE_BACKEND = "django-cache"

Is it possible to use django-db with chord tasks or it is not supported yet? If it is supported, how should I fix this error? Thanks a lot.

[ShuangLau]

Update:

I noticed that ChordCounter is missing in the django admin console. Does this mean anything?

[namper]

@ShuangLau Experience same problem.

From my analysis i found that chord fails when sub tasks results are too big. This can be checked by running

ChordCounter.objects.all().values()

We see existing counters not cleaned up and hence callback is not executed.

P.S ChordCounter doesn't have ModelAdmin and that's why it doesn't show, so that's normal.

[auvipy]

should that be shown in the admin?

[namper]

should that be shown in the admin?

No, because they are not persistent objects, as they are deleted after counter is zero

[auvipy]

then we probably should document this as intended behaviour? btw, any thought on improving the mechanism to handle bigger data?

[namper]

Agreed, we should document.

Not sure about solution.

You can have same problem with redis backend if result are too big, fetching them are very slow, and solution to that is to store result in storage and retrieve it from there. this approach can be used to side step limitation of database backend.