AppRegistryNotReady when trying to use CELERY_RESULT_BACKEND = 'django-db'

[immerrr]

Hi!

I'm experiencing an issue very similar to what was described in #11, but I can't find a misconfiguration. I have been following http://docs.celeryproject.org/en/latest/django/first-steps-with-django.html so far, and I have:

celery.py

import os
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')

from celery import Celery

app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')

app.autodiscover_tasks()


@app.task(bind=True)
def debug_task(self):
    print('Request: {0!r}'.format(self.request))

And then in settings I have:

INSTALLED_APPS = [
    ....
    'django_celery_results',
]

...

CELERY_RESULT_BACKEND = 'django-db'

I can run a worker successfully, and I can run a task from django shell. What I cannot do is, for example, to inspect the result of the task:

$ celery -A myapp result 3e5d00a5-f20e-4d1b-8996-eaade4b1b051
Traceback (most recent call last):
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/objects.py", line 42, in __get__
    return obj.__dict__[self.__name__]
KeyError: 'backend'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/immerrr/.conda/envs/myapp/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 480, in handle_argv
    return self.execute(command, argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 285, in run_from_argv
    sys.argv if argv is None else argv, command)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 368, in handle_argv
    return self(*args, **options)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/result.py", line 36, in run
    task_result = result_cls(task_id)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/result.py", line 93, in __init__
    self.backend = backend or self.app.backend
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/objects.py", line 44, in __get__
    value = obj.__dict__[self.__name__] = self.__get(obj)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/base.py", line 1182, in backend
    return self._get_backend()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/base.py", line 900, in _get_backend
    self.loader)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/backends.py", line 65, in by_url
    return by_name(backend, loader), url
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/app/backends.py", line 45, in by_name
    cls = symbol_by_name(backend, aliases)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/kombu/utils/imports.py", line 56, in symbol_by_name
    module = imp(module_name, package=package, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 978, in _gcd_import
  File "<frozen importlib._bootstrap>", line 961, in _find_and_load
  File "<frozen importlib._bootstrap>", line 950, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 655, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 678, in exec_module
  File "<frozen importlib._bootstrap>", line 205, in _call_with_frames_removed
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/backends/__init__.py", line 4, in <module>
    from .database import DatabaseBackend
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/backends/database.py", line 7, in <module>
    from ..models import TaskResult
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django_celery_results/models.py", line 17, in <module>
    class TaskResult(models.Model):
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/db/models/base.py", line 110, in __new__
    app_config = apps.get_containing_app_config(module)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/apps/registry.py", line 247, in get_containing_app_config
    self.check_apps_ready()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/django/apps/registry.py", line 125, in check_apps_ready
    raise AppRegistryNotReady("Apps aren't loaded yet.")
django.core.exceptions.AppRegistryNotReady: Apps aren't loaded yet.

It seems like for some reason the fixup doesn't fire before the backend is accessed.

If it is of any help, accessing the backend happens from here:

  File "/home/immerrr/.conda/envs/myapp/bin/celery", line 11, in <module>
    sys.exit(main())
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/__main__.py", line 14, in main
    _main()
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 326, in main
    cmd.execute_from_commandline(argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 488, in execute_from_commandline
    super(CeleryCommand, self).execute_from_commandline(argv)))
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 281, in execute_from_commandline
    return self.handle_argv(self.prog_name, argv[1:])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 480, in handle_argv
    return self.execute(command, argv)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/celery.py", line 412, in execute
    ).run_from_argv(self.prog_name, argv[1:], command=argv[0])
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 285, in run_from_argv
    sys.argv if argv is None else argv, command)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 368, in handle_argv
    return self(*args, **options)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/base.py", line 244, in __call__
    ret = self.run(*args, **kwargs)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/bin/result.py", line 36, in run
    task_result = result_cls(task_id)
  File "/home/immerrr/.conda/envs/myapp/lib/python3.6/site-packages/celery/result.py", line 93, in __init__
    self.backend = backend or self.app.backend

Here's my pip freeze:

amqp==2.1.4
anyjson==0.3.3
billiard==3.5.0.2
celery==4.0.2
Cerberus==1.1
click==6.7
coverage==4.3.4
decorator==4.0.11
Django==1.11.1
django-celery-results==1.0.1
djangorestframework==3.6.2
first==2.0.1
httmock==1.2.6
ipython==6.0.0
ipython-genutils==0.2.0
kombu==4.0.2
pexpect==4.2.1
pickleshare==0.7.4
pip-tools==1.9.0
prompt-toolkit==1.0.14
ptyprocess==0.5.1
pudb==2017.1.2
py==1.4.33
Pygments==2.2.0
pytest==3.0.7
pytest-cov==2.4.0
pytest-django==3.1.2
pytz==2017.2
requests==2.13.0
simplegeneric==0.8.1
six==1.10.0
traitlets==4.3.2
urwid==1.3.1
vine==1.1.3
wcwidth==0.1.7

[optiz0r]

I was just about to file a very similar bug for this myself, but for celery -A proj call proj.tasks.mytask. I'm not acessing any models from my tasks directly (though I am invoking call_command on haystack's rebuild_index which would be touching models). Running the call command with the django-db results backend causes AppRegistryNotReady, but commenting out this setting allows the job to run successfully.

[lmarvaud]

You forgot to setup django : (cf. http://stackoverflow.com/questions/24793351/django-appregistrynotready)

import os
import django
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
# Setup django project
django.setup()

# Setup celery
app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')

[immerrr]

Normally, django.setup() happens on module import as a fixup which end up calling this method.

The problem here is that there's a codepath that involves django-celery-results that imports and starts using django-related modules before the fixup is executed.

[olymk2]

I have the exact same issue currently on a minimally configured celery setup.

Its a public project so you can pull if this helps debug this, seems very simple to produce this issue and i have spent many hours tweaking to try and make it go away with little luck.

https://github.com/maidstone-hackspace/maidstone-hackspace-website

for me enabling this line is what produces the "Apps not loaded" error

https://github.com/maidstone-hackspace/maidstone-hackspace-website/blob/master/config/settings/common.py#L350

you should be able to pull and create .en from the sample file and docker-compose -fdev.yml up to bring up the project and see the error.

did anyone else solve this, I can found people who have reported this but no one with an actual fix.

whats stranger is a friends project with basically the exact same setup seems to just work.

[Tecktron]

It's always the stupid file permissions!

I would like to share that I too was experiencing the "App not loaded" issue, I had everything in place, it took me testing almost every configuration I could think of and a ton of frustration. In the end it turned out to be that because I was using file logging, the permissions on the log file were blocking the celery user I had setup. Strange that it was causing this apps error, but once I updated the file permissions everything worked.

Here's how I eventually figured it out (this may help someone else debug): I sudo'd in as my celery user (everything seemed to work fine as me, the owner and root) Tried and ran the multi start... failed (obviously) So, just using a single worker, I tried to load celery: /opt/myapp/.tox/py35/bin/celery worker -A myapp That failed. I then activated the virtual environment and ran celery worker -A myapp Failed. Next I ran python manage.py check Hey I got an error about the log file permissions. Updated the file permissions, and then ran this in reverse, manage.py check worked. Within the virtual environment, worked! Deactivated,worked! Multi, worked! Logged out, ran my service, it now worked!

Hoping this may help someone else.

[jedie]

I have the same problem.

Normally, django.setup() happens on module import as a fixup which end up calling this method.

Why not remove all the django hack and init celery in AppConfig.ready()

M current solution is this:

from celery import Celery, current_app
from django.apps import AppConfig


class FoobarAppConfig(AppConfig):

    def ready(self):
        celery_app = Celery(
            # Don't use 'celery.fixups.django:fixup' builtin fixup:
            fixups=[]
        )
        current_app.autodiscover_tasks()
        current_app.config_from_object('django.conf:settings', namespace='CELERY')

Note: I didn't create a instance with app = Celery()

[DylanYoung]

Why was this marked invalid? I'm still not able to execute celery -A app result TASK_ID

Is the correct solution to run setup in celery.py? Is this documented somewhere?

[ioreshnikov]

@DylanYoung it's not reflected anywhere in the documentation and I still experience the same problem four years later after the original report (running celery 5.2.1). The work-around for me was to force initialize django with django.setup() before I configure the app as suggested here

https://github.com/celery/django-celery-results/issues/20#issuecomment-301485364

@auvipy Can you please explain your decision to mark the issue as Closed while the bug is still present and there is no fix?

[auvipy]

@DylanYoung it's not reflected anywhere in the documentation and I still experience the same problem four years later after the original report (running celery 5.2.1). The work-around for me was to force initialize django with django.setup() before I configure the app as suggested here

#20 (comment)

@auvipy Can you please explain your decision to mark the issue as Closed while the bug is still present and there is no fix?

was closed based on https://github.com/celery/django-celery-results/issues/20#issuecomment-301485364 but if you have any improvement you can share that. django has evolved, celery too, so there might be area of improvement

[immerrr]

It is true that a lot has happened in the 4 years since the issue was opened, but it appears that the current versions of celery and django-celery-results the original issue is still happening. So in my understanding, right now this backend comes with an extra configuration requirement, that is to run django.setup() somewhere around the instantiation of the celery celery.App object.

Most of the people don't run into it, because celery worker & celery shell commands run app.loader.import_default_modules() method that invokes a builtin fixup that initialises django automatically, and it is the rare celery result or celery call commands that hit this undocumented behaviour.

Here there are several ways to improve the situation:

1. add the django.setup() requirement to this part of celery docs which is used as the reference documentation for configuring this extension in this repo's README
2. remove the requirement to add django.setup() to all user code by adding someting like if django.apps.not_configured: django.setup() right before the definition of the TaskResult model
3. add ctx.obj.app.loader.import_default_modules() line to celery call and celery result commands, and whatever other commands that may require accessing app.backend property

As a user, I would prefer 2 or 3, because it means less user code and less divergence from the default "celery way" that just works for the other backends. As a maintainer of a different open-source project, I would prefer 1 or 2, because 3 would most likely involve a discussion with the core celery project on the import/fixup behaviour that could take quite some time and energy to resolve. But you may have your own preference here, @auvipy.

[ioreshnikov]

@immerrr thanks for the explanation!

Indeed, I hit this problem when trying to test my installation by directly invoking a task with celery call. The unit tests for the same task were calling it through apply_async() with a worker running in a separate thread so I didn't see this issue before.

The second option sounds more future-proof to me. If the celery team decides to add another command they don't have to remember to import_default_modules() in there as well. But as you said, it's up to @auvipy to decide.

In any case, I think it's a good idea to reopen the issue and remove the "invalid" tag.

[zuggernautt]

You forgot to setup django : (cf. http://stackoverflow.com/questions/24793351/django-appregistrynotready)

import os
import django
from celery import Celery

# set the default Django settings module for the 'celery' program.
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'myapp.settings')
# Setup django project
django.setup()

# Setup celery
app = Celery('myapp')
app.config_from_object('django.conf:settings', namespace='CELERY')

django.setup() solved the same issue I was having.

[auvipy]

I am open to any contribution which eliminate this issue popping up again and again for the users

[lmarvaud]

I am open to any contribution which eliminate this issue popping up again and again for the users

Is it really a celery's bug ? Django raise the exception : AppRegistryNotReady, which is the case.

[immerrr]

@lmarvaud please, refer to my earlier comment, there is (was, at least) an automatic "fixup" that initializes Django automatically in some cases but not the other ones.

I would consider it a bug, or at least a caveat that should be mentioned in the doc explicitly

[brablc]

Another year later and another newbie struggling :-) I do not like calling django.setup() (it feels wrong) and calling celery_app.loader.import_default_modules() fixes the problem. Unfortunately I was not able to make it conditional (e.g. by checking celery_app.loader.worker_initialized), because it is executed very early (before the worker is instantiated).

celery_app = Celery("adm")
celery_app.config_from_object("django.conf:settings", namespace="CELERY")
celery_app.autodiscover_tasks()
celery_app.loader.import_default_modules()