Defectdojo Uploading a large number of findings causes error (a foreign key constraint fails)

[nedakheiri]

Bug description While uploading a large number of findings to defectdojo, the most of them are uploaded, however the action encounter an error and the reminder findings cannot upload. For example, I tried to upload a file that contained 1400 findings, but only 1100 findings were uploaded. I uploaded the file both manually and through this API: DEFECTDOJO_URL/api/v2/import-scan/

The error shown in logs is:

django.db.utils.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`defectdojo`.`dojo_test_import_finding_action`, CONSTRAINT `dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find` FOREIGN 
KEY (`finding_id`) REFERENCES `dojo_finding` (`id`))')

Steps to reproduce Steps to reproduce the behavior:

1. Create a json file with more than 1500 findings
2. Upload this file via “Import Scan results” menu in “Findings” section of a product

Expected behavior All findings should be upload successfully

Environment information

• DefectDojo version : v. 3.31.0 and v. 2.24.0

Logs

returned_columns = self._batched_insert(
^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1825, in _batched_insert
self._insert(
File "/usr/local/lib/python3.11/site-packages/django/db/models/query.py", line 1791, in _insert
return query.get_compiler(using=using).execute_sql(returning_fields)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/models/sql/compiler.py", line 1660, in execute_sql
cursor.execute(sql, params)
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 67, in execute
return self._execute_with_wrappers(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 80, in _execute_with_wrappers
return executor(sql, params, many, context)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 84, in _execute
with self.db.wrap_database_errors:
File "/usr/local/lib/python3.11/site-packages/django/db/utils.py", line 91, in __exit__
raise dj_exc_value.with_traceback(traceback) from exc_value
File "/usr/local/lib/python3.11/site-packages/django/db/backends/utils.py", line 89, in _execute
return self.cursor.execute(sql, params)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django_prometheus/db/common.py", line 69, in execute
return super().execute(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/django/db/backends/mysql/base.py", line 75, in execute
return self.cursor.execute(query, args)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 206, in execute
res = self._query(query)
^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/site-packages/MySQLdb/cursors.py", line 319, in _query
db.query(q)
File "/usr/local/lib/python3.11/site-packages/MySQLdb/connections.py", line 254, in query
_mysql.connection.query(self, query)
django.db.utils.IntegrityError: (1452, 'Cannot add or update a child row: a foreign key constraint fails (`defectdojo`.`dojo_test_import_finding_action`, CONSTRAINT `dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find` FOREIGN 
KEY (`finding_id`) REFERENCES `dojo_finding` (`id`))')

[kiblik]

Hi,

• Are you using DD_ASYNC_FINDING_IMPORT: True? Because this might be side-effect of this experimental functionality. If so, try to set it to False
• I see you are using quite old DD version, can you try to upgrade
  • Sidenote: Since 2.37.0, MySQL is not supported anymore. You will need to migrate to PostgreSQL

[valentijnscholten]

We believe this has been fixed recently. Can you try again with the latest version of Defect Dojo?

[github-actions[bot]]

This issue has been automatically closed because it was manually labeled as stale. If you believe this was closed in error, please reopen it and remove the stale label.