django-import-export-extensions icon indicating copy to clipboard operation
django-import-export-extensions copied to clipboard

PicklingError in case if ValidationError related to model constraints occur

Open yalef opened this issue 1 year ago • 1 comments

If Resource class has enabled clean_model_instances and error related to model field constraints occur during import, then unexpected PicklingError would be raised. Example:

#  model.py
class Artist(models.Model):
    """Model representing artist."""
    name = models.CharField(max_length=100, unique=True)
    bands = models.ManyToManyField("Band", through="Membership")

# resources.py
class SimpleArtistResource(CeleryModelResource):
    """Artist resource with simple fields."""
    filterset_class = ArtistFilterSet

    class Meta:
        model = Artist
        fields = [
            "id",
            "name",
        ]
        clean_model_instances = True

After that pass artist with existing name to import file. Following exception will occur:

Traceback (most recent call last): File "/home/yalef/Projects/django-import-export-extensions/tests/../import_export_extensions/models/import_job.py", line 337, in parse_data self.save( File "/home/yalef/Projects/django-import-export-extensions/tests/../import_export_extensions/models/import_job.py", line 217, in save super().save( File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/base.py", line 822, in save self.save_base( File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/base.py", line 909, in save_base updated = self._save_table( ^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/base.py", line 1036, in _save_table updated = self._do_update( ^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/base.py", line 1101, in _do_update return filtered._update(values) > 0 ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/query.py", line 1278, in _update return query.get_compiler(self.db).execute_sql(CURSOR) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1990, in execute_sql cursor = super().execute_sql(result_type) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1549, in execute_sql sql, params = self.as_sql() ^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/sql/compiler.py", line 1953, in as_sql val = field.get_db_prep_save(val, connection=self.connection) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/django/db/models/fields/init.py", line 1013, in get_db_prep_save return self.get_db_prep_value(value, connection=connection, prepared=False) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/picklefield/fields.py", line 205, in get_db_prep_value value = force_str(dbsafe_encode(value, self.compress, self.protocol, self.copy)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/yalef/Projects/django-import-export-extensions/.venv/lib/python3.12/site-packages/picklefield/fields.py", line 66, in dbsafe_encode value = dumps(value, protocol=pickle_protocol) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ _pickle.PicklingError: Can't pickle <function capfirst at 0x7efdc25fa340>: it's not the same object as django.utils.text.capfirst

yalef avatar Mar 29 '24 12:03 yalef

Error is raised because BaseJob.result field is PickledObjectField basically using python's pickle module. And pickle.dumps(ValidationError) is not working correctly with ValidationError's which related to model constraint validation. It may be somehow connected with following issues:

  • https://code.djangoproject.com/ticket/21555
  • https://github.com/python/cpython/issues/44791

yalef avatar Mar 29 '24 12:03 yalef