Task fail with ImportError: No module named 'django-db'

[mounirmesselmeni]

Current configuration

Celery report:

software -> celery:4.0.2 (latentcall) kombu:4.0.2 py:3.5.2 billiard:3.5.0.2 py-amqp:2.1.4 platform -> system:Linux arch:64bit, ELF imp:CPython loader -> celery.loaders.app.AppLoader settings -> transport:amqp results:django-db

CELERYD_HIJACK_ROOT_LOGGER: False CELERY_RESULT_BACKEND: 'django-db' CELERY_RESULT_PERSISTENT: True CELERY_IGNORE_RESULT: True CELERY_TASK_RESULT_EXPIRES: datetime.timedelta(7) CELERY_QUEUE_HA_POLICY: 'all' CELERY_BROKER_URL: 'amqp://project:********@*******:5672/project' CELERY_WORKER_DIRECT: False CELERY_TIMEZONE: 'Europe/Berlin' CELERY_SEND_EVENTS: False CELERY_TRACK_STARTED: True

Content of celery.py:

import os
from celery import Celery
from celery.schedules import crontab

os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'project.settings')

app = Celery('project')

app.config_from_object('django.conf:settings', namespace='CELERY')
app.autodiscover_tasks()

tasks.py:

@celery_app.task(autoretry_for=(Exception,), retry_kwargs={'max_retries': 5})
def revoke_x(name, id2, id2):
    return

Steps to reproduce

I'm not able to find out how to reproduce it

Expected behavior

Task running

Actual behavior

It's a weird behavior, as it's not failing always, I have many others tasks running with no problems, this problem happen sometimes but only with this task, and I'm not able to understand what's the source of the issue. Error happening before executing the tasks (When trying to send it to the queue) The call: revoke_x.apply_async(['param1', 1, 1])

traceback: ImportError: No module named 'django-db' KeyError 'backend' kombu/utils/objects.py in get at line 42

    def __get__(self, obj, type=None):
    if obj is None:
    return self
    try:
    return obj.__dict__[self.__name__]
    except KeyError:
    value = obj.__dict__[self.__name__] = self.__get(obj)
    return value
    def __set__(self, obj, value):

ImportError No module named 'django-db'

celery/app/backends.py in by_url at line 65 scheme, _, _ = url.partition('://') if '+' in scheme: backend, url = url.split('+', 1) else: backend = scheme return by_name(backend, loader), url

Values in context (From Sentry) backend: 'django-db' loader: <celery.loaders.app.AppLoader object at 0x7f13258a2f28> url: None

[georgepsarakis]

Have you installed the Python package django-celery-results, as stated in the documentation?

[mounirmesselmeni]

Yes it's installed and it's in the INSTALLED_APPS

pip freeze | grep celery

celery==4.0.2 django-celery-beat==1.0.1 django-celery-results==1.0.1

[atkawa7]

@MounirMesselmeni @georgepsarakis I had the same problem. Here is what I ended up adding

app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

[normic]

I stumbled about this issue today on our production system, but interestingly not on my dev machine. I was able to fix it by setting the option suggested by @atkawa7 .

To sum things up, I'm using: celery==4.0.0
django-celery-results==1.0.1

The only difference I could find within the virtualenvs is the used python version. Python 2.7.3 (prod - not working) Python 2.7.12 (dev - working)

And of course, I'm using Linux Mint on Dev and Debian Wheezy on the Production machin, but due to the use of virtualenvs this will problably not matter.

Maybe this helps to sort things out.

[auvipy]

@normic did you tried @atkawa7 solution?

[normic]

@auvipy Of course I did and it fixed my problem, I've written this in my comment (probably you overlooked the second sentence). I simply wrote it to help sort out things.

[peterfarrell]

Just wanted to confirm with celery 4.2.1 and django-celery-results (installed as master from github) we encountered this issue on one of boxes but not during local development.

Using the fix from @atkawa7 from above in our celery.py configuration file fixed the issue:

app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

[DZPM]

Hi, just got bit by this bug:

• happened in production, didn't notice in local
• used the fix from @atkawa7, successfully (Thanks!)

If the bug is still happening, I guess it shouldn't be closed yet?

[retosteffen]

I had the same issue today. It worked on a local dev machine but not on a cloud test one. @atkawa7 fix solves it. Thanks!

[washeck]

@auvipy Why was this issue closed? I understand the issue can be solved by add the above live (override_backends) but the documentation does not mention it.

So is it required to always the override and the documentation should be fixed? Or is it a bug in celery and the override is just a workaround?

[thedrow]

I honestly don't know if this issue was resolved. I'm not all that familiar with how the Django integration works.

[Kirade]

Hi, This also happend in my project. I upgrade celery version from 4.2.1 to 4.4.2

In my case, This happens while running by_name function in celery/app/backends.py.

# celery/app/backends.py 

BACKEND_ALIASES = {
    'amqp': 'celery.backends.amqp:AMQPBackend',
    'rpc': 'celery.backends.rpc.RPCBackend',
    'cache': 'celery.backends.cache:CacheBackend',
    'redis': 'celery.backends.redis:RedisBackend',
    'rediss': 'celery.backends.redis:RedisBackend',
    'sentinel': 'celery.backends.redis:SentinelBackend',
    'mongodb': 'celery.backends.mongodb:MongoBackend',
    'db': 'celery.backends.database:DatabaseBackend',
    'database': 'celery.backends.database:DatabaseBackend',
    'elasticsearch': 'celery.backends.elasticsearch:ElasticsearchBackend',
    'cassandra': 'celery.backends.cassandra:CassandraBackend',
    'couchbase': 'celery.backends.couchbase:CouchbaseBackend',
    'couchdb': 'celery.backends.couchdb:CouchBackend',
    'cosmosdbsql': 'celery.backends.cosmosdbsql:CosmosDBSQLBackend',
    'riak': 'celery.backends.riak:RiakBackend',
    'file': 'celery.backends.filesystem:FilesystemBackend',
    'disabled': 'celery.backends.base:DisabledBackend',
    'consul': 'celery.backends.consul:ConsulBackend',
    'dynamodb': 'celery.backends.dynamodb:DynamoDBBackend',
    'azureblockblob': 'celery.backends.azureblockblob:AzureBlockBlobBackend',
    'arangodb': 'celery.backends.arangodb:ArangoDbBackend',
    's3': 'celery.backends.s3:S3Backend',
}

aliases = dict(BACKEND_ALIASES, **loader.override_backends)
aliases.update(
        load_extension_class_names(extension_namespace) or {})
try:
    cls = symbol_by_name(backend, aliases)
except ValueError as exc:
    ....

Inteneded value of aliases should be BACKEND_ALIASES. As BACKEND_ALIASES defined just above the aliases assignment

But when exception raised, value of aliases was

{
    'amqp': 'celery.backends.amqp:AMQPBackend', 
    'cassandra': 'celery.backends.cassandra:CassandraBackend', 
    'consul': 'celery.backends.consul:ConsulBackend', 
    'cosmosdbsql': 'celery.backends.cosmosdbsql:CosmosDBSQLBackend', 
    'couchbase': 'celery.backends.couchbase:CouchbaseBackend', 
    'couchdb': 'celery.backends.couchdb:CouchBackend', 
    'elasticsearch': 'celery.backends.elasticsearch:ElasticsearchBackend', 
    'file': 'celery.backends.filesystem:FilesystemBackend', 
    'mongodb': 'celery.backends.mongodb:MongoBackend', 
    'sentinel': 'celery.backends.redis:SentinelBackend'
}

which is outdated. This might be existed in version 3.1 ( My Project migration history: 3.1.x -> 4.2.1 -> 4.4.2 )

For some reason, constant BACKEND_ALIASES value pointing outdated value.

Though @atkawa7 's solution works well, I'm trying to find exact cause of this problem. (Maybe Some kind of cache or import stuffs in my project are problem)

Any idea on this issue?

[auvipy]

I am moving this to django-celery-results

[auvipy]

I am not sure why it is not working.

[auvipy]

@AllexVeldman can you share you insight on this?

[auvipy]

CELERY_RESULT_BACKEND = django-db CELERY_RESULT_BACKEND_DB = postgresql://..

[auvipy]

closing in favor of https://github.com/celery/django-celery-results/issues/105

[AllexVeldman]

I have not seen this before, also #105 seems to be about the CELERY_RESULT_BACKEND_DB setting which I also don't set without any issues.

Does this happen on workers that have been running and executing tasks or only on newly created workers?

[APouzi]

@MounirMesselmeni @georgepsarakis I had the same problem. Here is what I ended up adding

app.loader.override_backends['django-db'] = 'django_celery_results.backends.database:DatabaseBackend'

You saved my week. I have been attempting to figure this issue out for days with a whole combination of attempts. Documentation and everything. If it isn't too much to ask, how did you figure this out? Pretty impressive man, again thanks.

[shoaibmalek21]

Yes, I figured out this issue with a django celery results package.

pip install django-celery-results