Cachalot breaks loading fixtures with no PK

[Finndersen]

Cachalot appears to impact fixture loading (for test cases) somehow, I think perhaps for fixtures with no PK specified, or with "pk": null.

Fixture loading works as expected with cachalot disabled.

python3.9, django 3.1.7

[Andrew-Chen-Wang]

I'm not sure what you mean by fixture loading. Are you talking about pytest with factory boy? Please provide a reproducible example in addition to the version of cachalot's you're using. Thanks!

[Finndersen]

Just loading fixtures during standard Django TestCases, or using loaddata management command.

I will need some time to try create a minimum reproducible example (will be away for the next week). But it appears to be only happening when loading multiple fixture files at once, loading a single file directly appears to work.

It appears to be related to fixtures with FK fields specifying the related model using its Natural Key value (not integer PK) The related model is not being loaded and so the lookup using the Natural Key is failing, for example this occurs when loading a User model fixture:

django.core.serializers.base.M2MDeserializationError: (DoesNotExist('Group matching query does not exist.'), ['MY_GROUP_NAME'])

However the MY_GROUP_NAME group model is defined in the same fixture file before User model is, and loading just that single fixture file by itself appears to work fine...

[Finndersen]

I'm using cachalot 2.5.1

[Finndersen]

Note i am also using diskcache.DjangoCache backend, and updating SUPPORTED_CACHE_BACKENDS to add it in to be allowed

[Andrew-Chen-Wang]

Interesting. An example isn't necessary. Can you post the full stack trace? If cachalot enabled is causing this, then it might have something to do with us loading your models. Have you tried placing cachalot in your INSTALLED_APPS at the very bottom?

[Finndersen]

Still occurs with cachalot at the bottom of INSTALLED_APPS. Here's the stack trace from running a custom management command which just calls loaddata command for every fixture of every app:

Traceback (most recent call last):
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 292, in deserialize_m2m_values
    values.append(m2m_convert(pk))
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 278, in m2m_convert
    return model._default_manager.db_manager(using).get_by_natural_key(*value).pk
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\contrib\auth\models.py", line 87, in get_by_natural_key
    return self.get(name=name)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\query.py", line 429, in get
    raise self.model.DoesNotExist(
django.contrib.auth.models.DoesNotExist: Group matching query does not exist.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 239, in save_deferred_fields
    values = deserialize_m2m_values(field, field_value, using, handle_forward_references=False)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 298, in deserialize_m2m_values
    raise M2MDeserializationError(e, pk)
django.core.serializers.base.M2MDeserializationError: (DoesNotExist('Group matching query does not exist.'), ['GROUP_NAME'])

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\management\management_command.py", line 34, in handle
    self.run(*args, **options)
  File "C:\Users\USERNAME\PycharmProjects\Project\project\core\management\commands\initialise_local_db.py", line 34, in run
    load_all_app_fixtures()
  File "C:\Users\USERNAME\PycharmProjects\Project\django_unchained\utils\misc.py", line 367, in load_all_app_fixtures
    call_command('loaddata', fixture, database=db, verbosity=verbosity)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 168, in call_command
    return command.execute(*args, **defaults)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 116, in loaddata
    obj.save_deferred_fields(using=self.using)
  File "C:\Users\USERNAME\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 241, in save_deferred_fields
    raise DeserializationError.WithData(e.original_exc, label, self.object.pk, e.pk)
django.core.serializers.base.DeserializationError: Group matching query does not exist.: (my_app.customuser:pk=13) field_value was '['GROUP_NAME']'

It appears its due to the Group fixture entry with name 'GROUP_NAME' not being loaded, but works fine with cachalot disabled. I think it's due to the fixture being defined with pk=null, because setting the pk to a valid number fixes it (but I don't want to define PKs for fixtures which can be identified with Natural Keys)

[Finndersen]

Looks like the issue can be recreated by trying to load a fixture which has a model with Natural Key support and no PK specified, and another model with a FK field to the first model, using Natural Key values to refer to the first fixture model. if PK is specified on the first model then it works fine. So it's something about not loading fixtures with no PK properly. Here's stack trace from loaddata command:

Traceback (most recent call last):
  File "C:\Users\USER\PycharmProjects\Project\project\core\manage.py", line 21, in <module>
    execute_from_command_line(sys.argv)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 401, in execute_from_command_line
    utility.execute()
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\__init__.py", line 395, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 330, in run_from_argv
    self.execute(*args, **cmd_options)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\base.py", line 371, in execute
    output = self.handle(*args, **options)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 72, in handle
    self.loaddata(fixture_labels)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 114, in loaddata
    self.load_label(fixture_label)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\management\commands\loaddata.py", line 181, in load_label
    obj.save(using=self.using)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\core\serializers\base.py", line 223, in save
    models.Model.save_base(self.object, using=using, raw=True, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 790, in save_base
    updated = self._save_table(
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 895, in _save_table
    results = self._do_insert(cls._base_manager, using, fields, returning_fields, raw)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\base.py", line 933, in _do_insert
    return manager._insert(
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\manager.py", line 85, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\query.py", line 1254, in _insert
    return query.get_compiler(using=using).execute_sql(returning_fields)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 37, in inner
    return original(compiler, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 113, in inner
    return original(write_compiler, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\models\sql\compiler.py", line 1397, in execute_sql
    cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 98, in execute
    return super().execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\cachalot\monkey_patch.py", line 137, in inner
    return original(cursor, sql, *args, **kwargs)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 66, in execute
    return self._execute_with_wrappers(sql, params, many=False, executor=self._execute)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 75, in _execute_with_wrappers
    return executor(sql, params, many, context)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\utils.py", line 90, in __exit__
    raise dj_exc_value.with_traceback(traceback) from exc_value
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\utils.py", line 84, in _execute
    return self.cursor.execute(sql, params)
  File "C:\Users\USER\AppData\Local\Programs\Python\Python39\lib\site-packages\django\db\backends\sqlite3\base.py", line 413, in execute
    return Database.Cursor.execute(self, query, params)
django.db.utils.IntegrityError: Problem installing fixture 'C:\Users\USER\PycharmProjects\Project\project\apps\my_app\fixtures\FixtureName.json': Could not load my_app.CustomUser(pk=None): NOT NULL constraint failed: my_app_customuser.foreign_key_field_id

[Finndersen]

@Andrew-Chen-Wang any luck with this?

[Andrew-Chen-Wang]

Hi @Finndersen i've been swamped with tests and work this last month, so i've been unable to find a time to try and resolve this. I'm not really sure if I'll be able to in the near future either, but this is still on my mind. Apologies in advance.

[Finndersen]

No worries, I could perhaps have a crack at it if I find some spare time, have you got any leads in mind?

On Mon, 11 Apr 2022, 9:58 pm Andrew Chen Wang, @.***> wrote:

Hi @Finndersen https://github.com/Finndersen i've been swamped with tests and work this last month, so i've been unable to find a time to try and resolve this. I'm not really sure if I'll be able to in the near future either, but this is still on my mind. Apologies in advance.

— Reply to this email directly, view it on GitHub https://github.com/noripyt/django-cachalot/issues/214#issuecomment-1094961868, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEEJXGNDQ75E3JYJQ5RRE6DVEQHW3ANCNFSM5QIWCFPQ . You are receiving this because you were mentioned.Message ID: @.***>

[Andrew-Chen-Wang]

One thing to note is whether you have the model manager set up correctly (I'm sure you do, but just double checking): https://stackoverflow.com/a/25493071

https://docs.djangoproject.com/en/4.0/topics/serialization/ (model manager portion)

[Finndersen]

I'm using django-natural-keys which should be managing all that fine

[Finndersen]

I'm using django-natural-keys but also the User model has built in natural key support

On Mon, 11 Apr 2022, 10:20 pm Andrew Chen Wang, @.***> wrote:

One thing to note is whether you have the model manager set up correctly (I'm sure you do, but just double checking): https://stackoverflow.com/a/25493071

— Reply to this email directly, view it on GitHub https://github.com/noripyt/django-cachalot/issues/214#issuecomment-1094981487, or unsubscribe https://github.com/notifications/unsubscribe-auth/AEEJXGJLEFGZM6CILQS7PLLVEQKJNANCNFSM5QIWCFPQ . You are receiving this because you were mentioned.Message ID: @.***>

[milos-u]

Maybe this is related: https://github.com/noripyt/django-cachalot/issues/224