Dangerous overflow in failed_import_rows

[bergstar]

Package

filament/filament

Package Version

v3.3.13

Laravel Version

v11.44.4

Livewire Version

No response

PHP Version

PHP 8.3.17

Problem description

Records in failed_import_rows reached well over a 7 million just after a single import. Import failed, but insane amount of records where created.

Expected behavior

A way to control / disable / prevent or minimise growth of the failed_import_rows. This is also undocumented behaviour which led to a serious situation in production. Garbage collection or some kind of handling of this must be in place.

Steps to reproduce

Create an import that failes and see the DB grow.

Reproduction repository (issue will be closed if this is not valid)

https://github.com/bergstar/app-filament-issue-3.x

Relevant log output

mysql -uaaaaaaa -p'aaaaaaa' \
  -e "SELECT table_schema          AS db_name,
             table_rows            AS rows_estimate
        FROM information_schema.tables
       WHERE table_name = 'failed_import_rows';"
mysql: [Warning] Using a password on the command line interface can be insecure.
+---------+---------------+
| db_name | rows_estimate |
+---------+---------------+
| aaaaaaa |       7893564 |
+---------+---------------+

[github-actions[bot]]

Hey @bergstar! We're sorry to hear that you've hit this issue. 💛

However, it looks like you forgot to fill in the reproduction repository URL. Can you edit your original post and then we'll look at your issue?

We need a public GitHub repository which contains a Laravel app with the minimal amount of Filament code to reproduce the problem. Please do not link to your actual project, what we need instead is a minimal reproduction in a fresh project without any unnecessary code. This means it doesn't matter if your real project is private / confidential, since we want a link to a separate, isolated reproduction. That would allow us to download it and review your bug much easier, so it can be fixed quicker. Please make sure to include a database seeder with everything we need to set the app up quickly.

[github-actions[bot]]

Thank you for providing reproduction steps! Reopening the issue now.

[danharrin]

How many rows were in your import CSV?

[danharrin]

Also there is no Filament import code and example import file in your reproduction repo, this is going to get closed if its not reproducible

[bergstar]

How many rows were in your import CSV?

File was 1.4mb so it was quite a few rows. I dont have the file since it happened in production with the customers data..

Also there is no Filament import code and example import file in your reproduction repo, this is going to get closed if its not reproducible

There is basic import CSV functionality as it is documented in the docs. Nothing special, nothing custom, 1:1 as it is in the docs.

[tonypartridge]

We understand what you are saying @bergstar but we need it as simple as possible to recreate, we have lots of issues to go through and adding 15 minutes onto a task is something we could do with saving, so the repo needs to recreate exactly the issue you are having.

[danharrin]

Were there 7 million rows in your import file? There should be one row for each failed row in the CSV. You can check this in the imports table, it has a total column.

If there is more than one row for each failure, that is a bug in Filament. If the original file contains the same number or more rows, then there is no issue in Filament. This table is how we collect issues with imports so that they can be collected and sent to the user in a report.

[bergstar]

Were there 7 million rows in your import file? There should be one row for each failed row in the CSV. You can check this in the imports table, it has a total column.

If there is more than one row for each failure, that is a bug in Filament. If the original file contains the same number or more rows, then there is no issue in Filament. This table is how we collect issues with imports so that they can be collected and sent to the user in a report.

Short answer is no, I checked with the customer, and it was 12000 lines in the CSV file. Size of the file was 1.4mb

There is no records of the failed upload in the imports table. or it got recorded as 1 line imported. maybe file missed line breaks?

Here is the dump of the imports table:

INSERT INTO xxx.imports (id, completed_at, file_name, file_path, importer, processed_rows, total_rows, successful_rows, user_id, created_at, updated_at) VALUES (1, '2025-04-24 00:07:21', 'example.csv', '/home/forge/xxx.xxx.xxx/releases/20250423210547/storage/app/private/livewire-tmp/0I8bRzSki5Rqpyv1iBmGWq3yCD07JT-metaZXhhbXBsZS5jc3Y=-.csv', 'App\\Filament\\Imports\\ProductImporter', 2, 2, 2, 1, '2025-04-24 00:07:18', '2025-04-24 00:07:21');
INSERT INTO xxx.imports (id, completed_at, file_name, file_path, importer, processed_rows, total_rows, successful_rows, user_id, created_at, updated_at) VALUES (2, '2025-04-24 09:45:36', 'v8_A0F1_e.csv', '/home/forge/xxx.xxx.xxx/releases/20250423210547/storage/app/private/livewire-tmp/9BC567xstD651t6a8EyGPm6t3rT36M-metadjhfQTBGMV9lLmNzdg==-.csv', 'App\\Filament\\Imports\\ProductImporter', 104, 104, 104, 3, '2025-04-24 09:45:34', '2025-04-24 09:45:36');
INSERT INTO xxx.imports (id, completed_at, file_name, file_path, importer, processed_rows, total_rows, successful_rows, user_id, created_at, updated_at) VALUES (3, '2025-04-24 09:51:30', 'v8_A0F1_e.csv', '/home/forge/xxx.xxx.xxx/releases/20250423210547/storage/app/private/livewire-tmp/WSQJKyoYVsXaoJHc61rdWeTeUXmUcz-metadjhfQTBGMV9lLmNzdg==-.csv', 'App\\Filament\\Imports\\ProductImporter', 104, 104, 104, 3, '2025-04-24 09:51:29', '2025-04-24 09:51:30');
INSERT INTO xxx.imports (id, completed_at, file_name, file_path, importer, processed_rows, total_rows, successful_rows, user_id, created_at, updated_at) VALUES (4, '2025-04-26 14:31:26', 'order_9de9bb03-8ecc-4cce-94f4-0ce4770d37f5_gtin_04640310330105_quantity_1.csv', '/home/forge/xxx.xxx.xxx/releases/20250425054423/storage/app/private/livewire-tmp/F573Bl6sygYG9fgsny2WAscnGUnCy4-metab3JkZXJfOWRlOWJiMDMtOGVjYy00Y2NlLTk0ZjQtMGNlNDc3MGQzN2Y1X2d0aW5fMDQ2NDAzMTAzMzAxMDVfcXVhbnRpdHlfMS5jc3Y=-.csv', 'App\\Filament\\Imports\\CodeImporter', 1, 1, 0, 3, '2025-04-25 14:31:26', '2025-04-26 14:31:26');

[bergstar]

We understand what you are saying @bergstar but we need it as simple as possible to recreate, we have lots of issues to go through and adding 15 minutes onto a task is something we could do with saving, so the repo needs to recreate exactly the issue you are having.

I understand, I will try to put together a test - but it will take sometime as week is busy.

[danharrin]

Were there any server exceptions originating from the import, or are the rows all validation related?

If there were server exceptions, maybe job retries kicked in. You could try using this method to turn off 24h retries (return null) - https://filamentphp.com/docs/3.x/actions/prebuilt-actions/import#customizing-the-import-job-retries

[danharrin]

I'm going to close this as there isn't really enough info for us to work with here. v4 has an alternative retry strategy for imports too, which I think could have been the culprit from v3