FactoryBoy
bug with django 4.1
Description
we are using factory boy with django unit test but when we upgraded from 4.0.4 to 4.1.1 i got the following errors
Traceback (most recent call last):
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 928, in get_or_create
return self.get(**kwargs), False
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/cacheops/query.py", line 353, in get
return qs._no_monkey.get(qs, *args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 650, in get
raise self.model.DoesNotExist(
src.core.timezone.models.TimeZone.DoesNotExist: TimeZone matching query does not exist.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL: Key (id)=(4) already exists.
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 143, in _get_or_create
instance, _created = manager.get_or_create(*args, **key_fields)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 935, in get_or_create
return self.create(**params), True
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 671, in create
obj.save(force_insert=True, using=self.db)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 831, in save
self.save_base(
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 882, in save_base
updated = self._save_table(
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 1025, in _save_table
results = self._do_insert(
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/base.py", line 1066, in _do_insert
return manager._insert(
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/manager.py", line 85, in manager_method
return getattr(self.get_queryset(), name)(*args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/query.py", line 1790, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/models/sql/compiler.py", line 1657, in execute_sql
cursor.execute(sql, params)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 103, in execute
return super().execute(sql, params)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/cacheops/transaction.py", line 98, in execute
result = self._no_monkey.execute(self, sql, params)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
django.db.utils.IntegrityError: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL: Key (id)=(4) already exists.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/atlassian/pipelines/agent/build/.venv/lib/python3.9/site-packages/django/test/testcases.py", line 1448, in setUpClass
cls.setUpTestData()
File "/opt/atlassian/pipelines/agent/build/adpp_backend/src/tests.py", line 47, in setUpTestData
cls._admin_user = cls._create_user()
File "/opt/atlassian/pipelines/agent/build/adpp_backend/src/tests.py", line 41, in _create_user
return AdminUserFactory()
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 40, in __call__
return cls.create(**kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 528, in create
return cls._generate(enums.CREATE_STRATEGY, kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 120, in _generate
return super()._generate(strategy, params)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 465, in _generate
return step.build()
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
step.resolve(pre)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
self.attributes[field_name] = getattr(self.stub, field_name)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
value = value.evaluate_pre(
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
return self.evaluate(instance, step, context)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
return step.recurse(subfactory, extra, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
return builder.build(parent_step=self, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
step.resolve(pre)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
self.attributes[field_name] = getattr(self.stub, field_name)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
value = value.evaluate_pre(
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
return self.evaluate(instance, step, context)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
return step.recurse(subfactory, extra, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
return builder.build(parent_step=self, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 260, in build
step.resolve(pre)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 201, in resolve
self.attributes[field_name] = getattr(self.stub, field_name)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 346, in __getattr__
value = value.evaluate_pre(
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 48, in evaluate_pre
return self.evaluate(instance, step, context)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/declarations.py", line 411, in evaluate
return step.recurse(subfactory, extra, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 218, in recurse
return builder.build(parent_step=self, force_sequence=force_sequence)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/builder.py", line 264, in build
instance = self.factory_meta.instantiate(
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/base.py", line 317, in instantiate
return self.factory._create(model, *args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 166, in _create
return cls._get_or_create(model_class, *args, **kwargs)
File "/opt/atlassian/pipelines/agent/build/.venv/src/factory-boy/factory/django.py", line 147, in _get_or_create
for lookup, value in cls._original_params.items()
AttributeError: type object 'TimeZoneFactory' has no attribute '_original_params'
To Reproduce
Share how the bug happened:
Model / Factory code
# Include your factories and models here
import factory
from factory.faker import faker
from .models import TimeZone
fake = faker.Faker()
class TimeZoneFactory(factory.django.DjangoModelFactory):
class Meta:
model = TimeZone
name = factory.Sequence(lambda n: f'{fake.name()}_{n}')
#models
class TimeZone(BaseModel):
"""
TimeZone
"""
class Meta:
db_table = 'core_timezone'
verbose_name = _('time zone')
verbose_name_plural = _('time zones')
name = models.TextField(_('name'), unique=True)
The issue
this factory is used everywhere in the project, not sure why this error appeared after upgrading to django 4.1.1
The package introduced support for django 4.1 but it was not published yet. Are there plans to publish a new version?
The step to reproduce lack details. For example, we see in the stack trace that get_or_create is used, but the factories from the STR do not use that feature. Please provide enough details to reproduce the issue. Ideally, a test case, or at least a minimal project where the issue can be reproduced.
Shot in the dark: did you override the _generate() hook in your factory and not call super()._generate()? The cls._original_params are defined there:
https://github.com/FactoryBoy/factory_boy/blob/8e5b79ab36b7918d723c382b3e6a59bca93d28b6/factory/django.py#L115-L120
@francoisfreitag Hi,
this Factory is a part of large code base, everything was working fine before upgrading to Django 4.1. I installed Django4.1.1 and run Django unit test as usual, i got those errors in the issue description.
NOTE: we didn’t override _generate()
i will try to provide a test case.
I get the same issue when going from Django 3.2.16 to 4.0.8, with factory_boy 3.2.1.
I have found the reason why I see this.
Django 3.2 allows accessing foreign keys before an object is saved for the first time. Django 4.0 does not. I worked around this by saving twice in save(), and that triggers the IntegrityError.
Example:
class Directory(models.Model):
name = models.CharField(max_length=255)
parent = models.ForeignKey('self', null=True)
num_subdirectories = models.IntegerField(default=0)
# works on 3.2
def save(self, *args, **kwargs):
self.num_subdirectories = self.directory_set.count()
super().save(*args, **kwargs)
# needed for 4.0
def save(self, *args, **kwargs):
if not self.id:
super().save(*args, **kwargs)
self.num_subdirectories = self.directory_set.count()
super().save(*args, **kwargs)
@hishamkaram @hmpf is there any chance your problem is solved with https://github.com/FactoryBoy/factory_boy/pull/981
Perhaps you can run you code against that branch?
@francoisfreitag _original_params errors disappeared but still getting this UniqueViolation error. I don’t pass any id, I am just using create_batch and SubFactory in several test cases
psycopg2.errors.UniqueViolation: duplicate key value violates unique constraint "core_timezone_pkey"
DETAIL: Key (id)=(1) already exists.
It looks like an issue with your factory. get_or_create lookup fails, tries to insert a new object that violates the unique constraint.
If you think this is a bug from factory boy, please provide a minimal project / script to reproduce the example, or (even better) a failing test in the test suite.
@francoisfreitag actually there is no get_or_create lookup in the factory but i will try to check how can i reproduce
Does your database have an autoincrement for TimeZone.id or is your sequence not in sync? It complains about that id=1 already exists and as far as I know Django does not generate primary keys but the database does.
@francoisfreitag @foarsitter true, thank you so much. i think this branch fixed the issue