Duplicates deletion error

[kokhanevych-macpaw]

Bug description I found the issue with duplicates deletion. When I enabled this feature I started to face such errors sometimes during report uploading: {"message":"Internal server error, check logs for details"} In django logs I found the error:

ERROR [dojo.api_v2.exception_handler:32] insert or update on table "dojo_test_import_finding_action" violates foreign key constraint "dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find"
DETAIL: Key (finding_id)=(742929) is not present in table "dojo_finding".
django.db.utils.IntegrityError: insert or update on table "dojo_test_import_finding_action" violates foreign key constraint "dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find"

As I understand DD tried to find a finding that was deleted as a duplicate.

@valentijnscholten commented on Slack: "The job to delete duplicates runs every minute, so if you have an import that runs longer than that, there may be a race condition that newly imported findings which are duplicates of existing findings are deleted before the import is complete. I can't remember exactly what the transactional model is in django, but probably not everything runs in a transaction so duplicates become visible to the job that deletes duplicates before the import is completed."

Steps to reproduce Steps to reproduce the behavior:

1. Create a project.
2. Enable Deduplicate findings and Delete duplicates features in settings.
3. Import a report with a lot of findings (more than 100).
4. Import a few reports with duplicates to the project.
5. After some imports you will face an error in logs: {"message":"Internal server error, check logs for details"}

Expected behavior The report should be imported successfully.

Deployment method (select with an X)

• [ ] Docker Compose
• [X] Kubernetes
• [ ] GoDojo

Environment information

• DefectDojo version 2.9.1

Logs

ERROR [dojo.api_v2.exception_handler:32] insert or update on table "dojo_test_import_finding_action" violates foreign key constraint "dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find"
DETAIL: Key (finding_id)=(742929) is not present in table "dojo_finding".
django.db.utils.IntegrityError: insert or update on table "dojo_test_import_finding_action" violates foreign key constraint "dojo_test_import_fin_finding_id_28fe8e2d_fk_dojo_find"