Problem Adding and Editing Parts in Large Projects

[T0b121]

Describe the bug When adding parts to a project in Part-DB, an error occurs after a certain number of entries. The issue manifests in the following scenarios:

1. Adding Parts: After a certain number of parts (tested in increments of 100 from 100 to 1000), no further parts can be added.
2. Editing Projects: After a certain number of parts (e.g., 200), the project can no longer be moved or edited. The error "CSRF token is invalid" occurs.
3. Moving Projects: When moving projects to another folder, the part names are deleted, and the error "CSRF token is invalid" occurs.
4. Importing Parts: When importing parts into projects with 600 or more items, an error occurs. The projects can be viewed in the project tab but cannot be edited via the edit tab.
5. Demo Website: The same error occurs on the demo website (https://demo.part-db.de/en/project/187/edit#bom) when editing a project with a large number of parts.

To Reproduce Problem with Edit of a Project

1. Create a new project in Part-DB.
2. Import KiCAD BOM with many parts (attached are BOM with a multiple of 100: 100, 200, 300, ..., 1000).
3. Attempt to add more parts, move Project ore edit it over all when it has 200 or more parts. Edit Page is not more able to open
4. Import parts into a project with 600 or more items.

Expected behavior

• Parts should be added without issues, regardless of the number.
• Projects should be editable and movable, regardless of the number of parts.
• Parts import should work regardless of the number of items in the project.
• The demo website should function correctly when editing projects with a large number of parts.

Screenshots

• Image 1_Project with 200 Parts befor move.png
• Image 2_Project with 200 Parts after move.png
• Image 3_Open Edit from Project with 600 Parts.png

Server Side

• Docker 20.10.24+dfsg1
  • Part-DB Version: [1.17.0]
  • Database Server [MariaDB Community 11.7.2]

Desktop:

• OS: [Windows 11]
• Browser [Edge]
• Version [135.0.3179.54 64-Bit]

Additional context PartDB config PartDB setting.zip

[d-buchmann]

Note: The server responds with code 422 (unprocessable content). ~~The POST is >800kB and chunked, however nowhere near the php post_max_size limit on my instance on which I reproduced it.~~ When inspecting the POST body, I found that the bom_entries's IDs are messed up from item 142 onwards (6 alphanumeric chars instead of db_<number>). That's obviously causing the 422 and thus the (a little misleading) error message. I read somewhere that certain multipart decoders have an arbitrary parameter limit of 1000, which is just hit here when adding the 142th bom entry.

[d-buchmann]

You say

should work regardless of the number of items

which is not going to happen, because any user could DOS your server. They certainly won't, but it's a vulnerability that the lib designers already thought of and prevented. Of course there are ways to increase the limit, but not infinitely.

Theoretically, possible solutions/workarounds are:

• Merge the fields of each BOM entry into a JSON string and use that in the multipart transfer
• Looping POSTs until the list is walked through, making the DB access non-atomic and violating statefulness(?)
• Split your projects into subprojects of BOM size <= 140

Could you try (untested, use at your own risk!)

sed -i 's/parameterlimit=1e3/parameterlimit=1e4/' vendor/api-platform/core/src/Symfony/Bundle/Resources/public/swagger-ui/swagger-ui-bundle.js
yarn build

and see if that at least solves the first issue?

BTW, what are you designing? ATCA cards? EATX Motherboards? =)

[T0b121]

I tested the recommended workaround (adjusting the parameterlimit in the swagger-ui-bundle.js). Unfortunately, this did not resolve our issue — the behavior remains unchanged.

Regarding what we design: We create complex analog circuits, for example for noise measurements, using a wide variety of specialized components: • Resistors with tolerances of 1 %, 0.1 %, or 0.05 % • Resistors with specific temperature coefficients     • Capacitors with different dielectric types (e.g. NP0, X7R)  

As a result, we can end up with over 142 different components in a single design.