django-cms icon indicating copy to clipboard operation
django-cms copied to clipboard
verified publisher icon django-cms

invalidate_cms_page_cache() prevents start, if db cachetable is non existent

Open adrian-stern opened this issue 10 years ago • 8 comments

Precondition: Django CMS App, working, without any Caching.

  1. Add simple db cache to settings.py
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'my_cache_table',
    }
}
  1. run manage.py createcachetable

This will result in the error shown in the stacktrace below.

CMS tries to invalidate the cache before the cachetable could be created.

As a workaround it is necessary to:

  1. remove all non basic django.* modules from INSTALLED_APPS
  2. run manage.py createcachetable
  3. readd all the modules again

Stack Trace:

./manage.py createcachetable /usr/local/lib/python3.4/importlib/_bootstrap.py:321: RemovedInDjango19Warning: django.utils.importlib will be removed in Django 1.9. return f(_args, *_kwds)

/usr/local/lib/python3.4/importlib/_bootstrap.py:321: RemovedInDjango19Warning: django.contrib.contenttypes.generic is deprecated and will be removed in Django 1.9. Its contents have been moved to the fields, forms, and admin submodules of django.contrib.contenttypes. return f(_args, *_kwds)

Traceback (most recent call last): File "/usr/local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) psycopg2.ProgrammingError: FEHLER: Relation „db_cache_table“ existiert nicht LINE 1: SELECT cache_key, value, expires FROM "db_cache_table" WHERE... ^

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "./manage.py", line 10, in execute_from_command_line(sys.argv) File "/usr/local/lib/python3.4/site-packages/django/core/management/init.py", line 351, in execute_from_command_line utility.execute() File "/usr/local/lib/python3.4/site-packages/django/core/management/init.py", line 325, in execute django.setup() File "/usr/local/lib/python3.4/site-packages/django/init.py", line 18, in setup apps.populate(settings.INSTALLED_APPS) File "/usr/local/lib/python3.4/site-packages/django/apps/registry.py", line 115, in populate app_config.ready() File "/usr/local/lib/python3.4/site-packages/django/contrib/admin/apps.py", line 22, in ready self.module.autodiscover() File "/usr/local/lib/python3.4/site-packages/django/contrib/admin/init.py", line 24, in autodiscover autodiscover_modules('admin', register_to=site) File "/usr/local/lib/python3.4/site-packages/django/utils/module_loading.py", line 74, in autodiscover_modules import_module('%s.%s' % (app_config.name, module_to_search)) File "/usr/local/lib/python3.4/importlib/init.py", line 109, in import_module return _bootstrap._gcd_import(name[level:], package, level) File "", line 2254, in _gcd_import File "", line 2237, in _find_and_load File "", line 2226, in _find_and_load_unlocked File "", line 1200, in _load_unlocked File "", line 1129, in _exec File "", line 1471, in exec_module File "", line 321, in _call_with_frames_removed File "/usr/local/lib/python3.4/site-packages/cms/admin/init.py", line 11, in plugin_pool.plugin_pool.discover_plugins() File "/usr/local/lib/python3.4/site-packages/cms/plugin_pool.py", line 32, in discover_plugins invalidate_cms_page_cache() File "/usr/local/lib/python3.4/site-packages/cms/cache/init.py", line 69, in invalidate_cms_page_cache version = _get_cache_version() File "/usr/local/lib/python3.4/site-packages/cms/cache/init.py", line 15, in _get_cache_version version = cache.get(CMS_PAGE_CACHE_VERSION_KEY) File "/usr/local/lib/python3.4/site-packages/django/core/cache/backends/db.py", line 66, in get "WHERE cache_key = %%s" % table, [key]) File "/usr/local/lib/python3.4/site-packages/django/db/backends/utils.py", line 79, in execute return super(CursorDebugWrapper, self).execute(sql, params) File "/usr/local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) File "/usr/local/lib/python3.4/site-packages/django/db/utils.py", line 97, in exit six.reraise(dj_exc_type, dj_exc_value, traceback) File "/usr/local/lib/python3.4/site-packages/django/utils/six.py", line 658, in reraise raise value.with_traceback(tb) File "/usr/local/lib/python3.4/site-packages/django/db/backends/utils.py", line 64, in execute return self.cursor.execute(sql, params) django.db.utils.ProgrammingError: FEHLER: Relation „db_cache_table“ existiert nicht LINE 1: SELECT cache_key, value, expires FROM "db_cache_table" WHERE...

adrian-stern avatar Mar 09 '16 15:03 adrian-stern

I was going to open an issue to raise this same bug, but then I discovered this one. I have created a test project to demonstrate this bug here. Attempting to do a python manage.py createcachetable raises the following exception:

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/core/management/__init__.py", line 328, in execute
    django.setup()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/apps/registry.py", line 115, in populate
    app_config.ready()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/contrib/admin/apps.py", line 22, in ready
    self.module.autodiscover()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/utils/module_loading.py", line 74, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/usr/local/Cellar/python/2.7.11/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/cms/admin/__init__.py", line 11, in <module>
    plugin_pool.plugin_pool.discover_plugins()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/cms/plugin_pool.py", line 31, in discover_plugins
    invalidate_cms_page_cache()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/cms/cache/__init__.py", line 69, in invalidate_cms_page_cache
    version = _get_cache_version()
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/cms/cache/__init__.py", line 15, in _get_cache_version
    version = cache.get(CMS_PAGE_CACHE_VERSION_KEY)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/core/cache/backends/db.py", line 66, in get
    "WHERE cache_key = %%s" % table, [key])
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/db/backends/utils.py", line 79, in execute
    return super(CursorDebugWrapper, self).execute(sql, params)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/db/utils.py", line 98, in __exit__
    six.reraise(dj_exc_type, dj_exc_value, traceback)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/db/backends/utils.py", line 64, in execute
    return self.cursor.execute(sql, params)
  File "/Users/eric/.virtualenvs/test/lib/python2.7/site-packages/django/db/backends/sqlite3/base.py", line 318, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.OperationalError: no such table: django_cache_table

Removing cms from INSTALLED_APPS causes the problem to disappear.

ericamador avatar Jun 21 '16 02:06 ericamador

@mkoistinen any insight from your end?

FinalAngel avatar Jun 21 '16 07:06 FinalAngel

run into the same Problem :(

my hacked work-a-round:

import sys
if "createcachetable" in sys.argv:
    INSTALLED_APPS = ()

jedie avatar Jul 20 '16 14:07 jedie

Has this been investigated further yet? This completely prevents the use of django.core.cache.backends.db.DatabaseCache and I've confirmed this is still a bug in django-cms v3.3.1.

ericamador avatar Jul 24 '16 23:07 ericamador

Along the same lines, if the cache is not reachable, a ConnectionError will be raised. If one part of the app uses django-cms (and therefore this code is run during Django's init), other parts of the app that don't use the CMS nor the cache will fail to run because the ConnectionError will crash the init sequence. Take for example reports being run by cron using Django's management command system.

In my case, a management command that doesn't use the CMS nor the cache crashes like such:

Traceback (most recent call last):
  File "./manage.py", line 16, in <module>
    execute_from_command_line(sys.argv)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 354, in execute_from_command_line
    utility.execute()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/core/management/__init__.py", line 328, in execute
    django.setup()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/__init__.py", line 18, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/apps/registry.py", line 115, in populate
    app_config.ready()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/contrib/admin/apps.py", line 22, in ready
    self.module.autodiscover()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/contrib/admin/__init__.py", line 24, in autodiscover
    autodiscover_modules('admin', register_to=site)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/django/utils/module_loading.py", line 74, in autodiscover_modules
    import_module('%s.%s' % (app_config.name, module_to_search))
  File "/usr/local/Cellar/python/2.7.12/Frameworks/Python.framework/Versions/2.7/lib/python2.7/importlib/__init__.py", line 37, in import_module
    __import__(name)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/cms/admin/__init__.py", line 11, in <module>
    plugin_pool.plugin_pool.discover_plugins()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/cms/plugin_pool.py", line 31, in discover_plugins
    invalidate_cms_page_cache()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/cms/cache/__init__.py", line 69, in invalidate_cms_page_cache
    version = _get_cache_version()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/cms/cache/__init__.py", line 15, in _get_cache_version
    version = cache.get(CMS_PAGE_CACHE_VERSION_KEY)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis_cache/backends/base.py", line 32, in wrapped
    return method(self, client, key, *args, **kwargs)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis_cache/backends/base.py", line 245, in get
    value = client.get(key)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis/client.py", line 863, in get
    return self.execute_command('GET', name)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis/client.py", line 570, in execute_command
    connection.send_command(*args)
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis/connection.py", line 556, in send_command
    self.send_packed_command(self.pack_command(*args))
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis/connection.py", line 532, in send_packed_command
    self.connect()
  File "/virtualenvs/reportsapp/lib/python2.7/site-packages/redis/connection.py", line 436, in connect
    raise ConnectionError(self._error_message(e))
redis.exceptions.ConnectionError: Error 61 connecting to 127.0.0.1:57842. Connection refused.

Line 15 in _get_cache_version() should handle cache errors or they should be handled higher up the stack.

heliodor avatar Feb 16 '17 17:02 heliodor

I'm just posting here to note that the correct solution to this bug (and related bugs) is almost certainly to delay hitting the cache until after app load. There are a number of reasons that hitting the cache too early is a bad idea.

julianandrews avatar Mar 17 '17 23:03 julianandrews

This Bug still exists with: Django-CMS v3.4.5 :(

A better work-a-round:

import sys
if "createcachetable" in sys.argv:
    INSTALLED_APPS = list(INSTALLED_APPS)
    INSTALLED_APPS.remove("cms")

Possible error by the work-a-round from above ( https://github.com/divio/django-cms/issues/5079#issuecomment-233961768 ) is:

RuntimeError: Model class django.contrib.sites.models.Site doesn't declare an explicit app_label and isn't in an application in INSTALLED_APPS.

jedie avatar Feb 23 '18 20:02 jedie

#7269 may provide a workaround for this

ipmb avatar Mar 31 '22 16:03 ipmb