Tree broken after node delete/change/add

[marcoacierno]

Summary

When the admin tries (for example) to delete a placeholder the delete fails with an error 500 saying MultipleObjectsReturned: get() returned more than one CMSPlugin -- it returned 36.

Traceback (most recent call last):
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/core/handlers/base.py", line 149, in get_response
    response = self.process_exception_by_middleware(e, request)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/core/handlers/base.py", line 147, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/utils/decorators.py", line 149, in _wrapped_view
    response = view_func(request, *args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/views/decorators/cache.py", line 57, in _wrapped_view_func
    response = view_func(request, *args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/contrib/admin/sites.py", line 244, in inner
    return view(request, *args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/views/decorators/clickjacking.py", line 39, in wrapped_view
    resp = view_func(*args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/cms/admin/placeholderadmin.py", line 678, in delete_plugin
    plugin.delete()
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/cms/models/pluginmodel.py", line 483, in delete
    super(CMSPlugin, self).delete(*args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/treebeard/models.py", line 506, in delete
    self.__class__.objects.filter(pk=self.pk).delete()
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/treebeard/mp_tree.py", line 112, in delete
    parents[parentpath] = node.get_parent(True)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/treebeard/mp_tree.py", line 1072, in get_parent
    self.__class__).objects.get(path=parentpath)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/db/models/manager.py", line 122, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/Users/marco/.virtualenvs/rollsite-2/lib/python2.7/site-packages/django/db/models/query.py", line 391, in get
    (self.model._meta.object_name, num)
MultipleObjectsReturned: get() returned more than one CMSPlugin -- it returned 35!

The error happens in MP_NodeQuerySet#delete (of django-treebeard) when the code tries to get a node parent with node.get_parent(True).

Looking at the database I can see that I have multiple plugins with the same path value (https://marcoacierno.space/xQzR3Qogo4.png) so I'm not sure but it might be the reason...

The "same path value for multiple plugins" also causes another bug in the delete code: when the deleted node is not a leaf node, most of the content in the website is deleted (because toremove.append(Q(path__startswith=node.path)) is used, which basically selects all my plugins in the database, even if in a different post)

Any suggestion how to debug the issue? Is normal that multiple plugins in the database have the same node path even when in different posts?

Commands run to fix/check what is happing:

• cms fix-tree fails with treebeard.exceptions.NodeAlreadySaved: Attempted to add a tree node that is already in the database
• cms check: https://pastebin.com/rYGWxmhq

The django-treebeard code:

# ok, got the minimal list of nodes to remove...
# we must also remove their children
# and update every parent node's numchild attribute
# LOTS OF FUN HERE!
for path, node in removed.items():
    parentpath = node._get_basepath(node.path, node.depth - 1)
    if parentpath:
        if parentpath not in parents:
            parents[parentpath] = node.get_parent(True)
        parent = parents[parentpath]
        if parent and parent.numchild > 0:
            parent.numchild -= 1
            parent.save()
    if node.is_leaf():
        toremove.append(Q(path=node.path))
    else:
        toremove.append(Q(path__startswith=node.path))

Expected behaviour

The plugin is correctly deleted and the site does not lose content.

Actual behaviour

When the admin tries to add/remove content there is a possibility that most of it gets deleted

Environment

• Python version: 2.7
• Django version: 1.9.5
• django CMS version: 3.2.5

[czpython]

Hello @marcoacierno, Thanks for reporting this. Which database are you using? The path field is a unique field, which means that your database is not handling constraints properly.

Also you're using a really old django CMS version, please upgrade to the latest django CMS 3.4.x

[marcoacierno]

Hello!

I'm using MySQL and, for some reason, in my database cms_cmsplugin#path is not unique.

I need to check if I can upgrade to 3.4.x, as it requires a new version of Django and I have a few libraries that are not compatible (so it might require a bit of time)

[czpython]

@marcoacierno I see you're using django 1.9 which is compatible with django cms 3.4.x

[marcoacierno]

@czpython Do you happen to know if cms fix-tree should fix the issue (the not unique path) I'm having or it's for something totally different? I upgraded to 3.4 but I can't test before I fix other issue (for some reason Django CMS can't find my Apphook (or so it seems) anymore so it doesn't register the namespace)

So I will let you know as soon as possibile

[czpython]

@marcoacierno rename any cms_app file to cms_apps. fix-tree won't fix the path issue, that said cms 3.4.x fixed a lot of tree corruptions. can you check if any unique constraint is enforced on your db?

[marcoacierno]

Done (renamed cms_toolbar), but now my entire site is blank. Is there any other compatibility change to do that is not specified in the changelogs?

I'm checking the changelogs of all the versions, all my custom plugins already use 'render' so I think the 'Manual plugin rendering' section does not apply to me. I'm having a look at the other external plugins I use and check if I need to update them (I'm sure I have to).

I also tried to recreate from 0 the database and reapply the migrations without importing the dump from the site, and I can see that path is still not unique (Update: Totally my fault, path is indeed unique). About your question with a quick look there isn't any field with the unique on it (can't confirm for sure for now, I need to do a migration that applies an unique to test or check what the migration says with what is in the database.)

I will write again soon with a better answer for the 'unique constraint' question

[marcoacierno]

can you check if any unique constraint is enforced on your db?

The other unique constraints seems to work fine. Only path, for some reason, isn't...

[czpython]

Was this project recently upgraded from a django version lower than 1.7?

[marcoacierno]

Hello, sorry for the silence.

It's not confirmed yet, but I think I fixed the issue by changing how django-treebeard selects the last root node.

The problem:

When django-treebeard called get_last_root_node in (for example) https://github.com/django-treebeard/django-treebeard/blob/2a7db13fc1d0738ab72bc2c28de1913727eecd47/treebeard/mp_tree.py#L315 to get the last root node path created, my database always returned 999 as latest root path (the function basically does a order_by('path') and reverse to get the last node), even if 099A was present.

I fixed it by ordering the roots using the id instead of the path https://github.com/rollstudio/django-treebeard/commit/00e4fe93afc14274c625b648945f31b5b0fec9ac

Basically, this means that ALL the root nodes created since 999 became the last root path, shared the same root path as the code always thought that 099A was the next "unique" path.

Then, I created this command to fix my broken paths in the database. I don't think this command will help someone else because it's very specific to my problem (and I'm pretty sure, if you database is in healthy state (I mean, path is marked as unique), you don't have duplicate paths or similar, as the creation of the plugin will very likely just fail with 'integrity error, path is not unique')

The data corruption doesn't seem to happen anymore, but we need to do more tests. I also haven't upgraded Django/django-cms as it caused too many problems and fixing this issue was more important.

I'm closing this issue as it doesn't seem to be related to django-cms. Thank you for your help @czpython !

[czpython]

Thanks for the update @marcoacierno. I suggest to have a look at the collation used on your database. Ordering by id will likely break your tree.

[marcoacierno]

It orders by id only the root nodes (so depth=1) not the entire tree

Will have a look when I arrive at the office

[jiangpuxuan]

I encountered the same problems when I tried to publish my edit. Quite annoying.

[jiangpuxuan]

I encountered the same problems when I tried to publish my edit. Quite annoying.

apache2.4, mod_wsgi 4.8, django 3.1.7, djangocms 3.8.0, ubuntu 20.04

[preinhart]

I have this problem as soon as I fill placeholders with plugins:

Attempted to add a tree node that is already in the database

Pages without plugins can be published, only with plugins produces the error, no matter which plugins.

django_cms==3.8.0, django-treebeard==4.5.1, Python==3.6.9, django==3.1.8

[marcoacierno]

While I don't work at the company where I had this issue anymore and I also know they stopped using django-cms due this problem (which was: every time they tried to change something it would delete everything, making updating the site impossible) and many other problems, as other people are reporting that they have the same issue I think reopening this issue might be helpful

Feel free to close it again

[preinhart]

For me, this problem occurs in general, after fresh installation on the first page.

Attempted to add a tree node that is already in the database Request Method: POST Request URL: XXX Django Version: 3.1.8 Exception Type: NodeAlreadySaved Exception Value: Attempted to add a tree node that is already in the database Exception Location: /python-app-venv/lib/python3.6/site-packages/treebeard/mp_tree.py, line 326, in process Python Executable: /python-app-venv/bin/python Python Version: 3.6.9

I have already tried different django_treebeard versions and also python manage.py cms check does not return any error.

[vinitkumar]

@preinhart Please check this thread, https://django-cmsworkspace.slack.com/archives/CQ97U91UM/p1613655590140300. The gist is that you need to pin django-treebeard to a version less than v4.5.0. Please read this thread for more details.

[preinhart]

Unfortunately no improvement with django_treebeard==4.4

DATABASES = {
  'default': {
       'CONN_MAX_AGE': 0,
       'ENGINE': 'django.db.backends.mysql',
       'OPTIONS': {
           'sql_mode': 'traditional',
       },
       'HOST': '',
       'NAME': '',
       'PASSWORD': '',
       'PORT': '',
       'USER': ''
   }
}

I am trying to configure two sites with one database, the first settings.py with SITE_ID = 1 and the second settings.py with SITE_ID = 2.

[vinitkumar]

Unfortunately no improvement with django_treebeard==4.4

DATABASES = {
  'default': {
       'CONN_MAX_AGE': 0,
       'ENGINE': 'django.db.backends.mysql',
       'OPTIONS': {
           'sql_mode': 'traditional',
       },
       'HOST': '',
       'NAME': '',
       'PASSWORD': '',
       'PORT': '',
       'USER': ''
   }
}

I am trying to configure two sites with one database, the first settings.py with SITE_ID = 1 and the second settings.py with SITE_ID = 2.

That's weird. Could you please post the output for your pip freeze?

[vinitkumar]

@preinhart

I have this problem as soon as I fill placeholders with plugins:

Attempted to add a tree node that is already in the database

Pages without plugins can be published, only with plugins produces the error, no matter which plugins.

django_cms==3.8.0, django-treebeard==4.5.1, Python==3.6.9, django==3.1.8

I can spot the problem right here. You are using django-treebeard==4.5.1. This is a buggy version. Switch to a version below 4.5 like this: django-treebeard==4.4 And you will not face these issues anymore.

[preinhart]

Both environments (SITE_ID = 1 and SITE_ID = 2) are equal, but not the same.

asgiref==3.3.4
dj-database-url==0.5.0
Django==3.1.8
django-classy-tags==2.0.0
django-cms==3.8.0
django-filer==2.0.2
django-formtools==2.2
django-js-asset==1.2.2
django-mptt==0.12.0
django-polymorphic==3.0.0
django-sekizai==2.0.0
django-treebeard==4.4
djangocms-admin-style==2.0.2
djangocms-attributes-field==2.0.0
djangocms-bootstrap4==2.0.0
djangocms-file==3.0.0
djangocms-googlemap==2.0.0
djangocms-icon==2.0.0
djangocms-installer==2.0.0
djangocms-link==3.0.0
djangocms-picture==3.0.0
djangocms-style==3.0.0
djangocms-text-ckeditor==4.0.0
djangocms-video==3.0.0
easy-thumbnails==2.7.1
html5lib==1.1
mysqlclient==2.0.3
Pillow==8.2.0
pytz==2021.1
six==1.15.0
sqlparse==0.4.1
typing-extensions==3.7.4.3
tzlocal==2.1
Unidecode==1.1.2
webencodings==0.5.1

Using django-treebeard==4.4 in both environments resulted in SITE_ID = 1 working, SITE_ID = 2 unfortunately still not.

I double checked that both environments use django-treebeard==4.4 :-)

Thanks for the help!

[preinhart]

My tests have shown that the problem:

Attempted to add a tree node that is already in the database

occurs regularly in a multisite setup. With one site SITE_ID = 1 it works without error, the error occurs as soon as a second environment with a new settings.py (SITE_ID = 2) uses the same database.

Maybe there is a way to set the SITE_ID in the settings.py dynamically? I tried something like this in my passenger_wsgi.py:

if os.getenv('SSL_TLS_SNI') == 'XY':
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', ApplicationName + '.settings_XY')
else:
    os.environ.setdefault('DJANGO_SETTINGS_MODULE', ApplicationName + '.settings_AB')

Does anyone knows a best practice to setup django_cms for multisites?

[vinitkumar]

@preinhart We use this at work. https://github.com/nephila/djangocms-multisite. This might solve your issues.

[marksweb]

@vinitkumar @preinhart On that note, I have a fork to fix migration from an empty database on the underlying library for that package;

https://github.com/shestera/django-multisite/compare/master...marksweb:master

I think I recall getting a Site does not exist type error when starting a new project with multisite support.

[greengoaxe]

Hi @preinhart, we need the code where we can reproduce the error and then try to fix it. Thanks for your cooperation. greengoaxe