RupertAvery
[BUG] Crashing when too much metadata in one update
Describe the bug If there is too much metadata in a single update to the database, the app crashes. This can be achieved by trying to scan a batch of comfyui images with large workflows. Scanning them in piecemeal works fine, but if they are all scanned at once the toolkit crashes.
Version: 1.8
Tool: ComfyUI
To Reproduce Take 100 or so unique images all with very large amounts of raw metadata, around 50k characters apiece. Put them all in one folder. Try to scan that folder. The app crashes with no message.
Expected behavior It should scan.
Additional context Again, none of these images even with 50k metadata characters seem to be an issue if scanned in a few at a time. It's only when the scanner tries to ingest a lot at once that it crashes. I suspect it's overflowing something somewhere.
Hmmm I do try to batch Node/NodeProperty updates to the database for every 100 images, so if an image has hundreds of nodes and thousands of properties that might overwhelm the batch updater.
You are scanning with Store Metadata and Store Comfy Workflows enabled?
Can you post a sample image here so I can estimate the batch size I need to limit it to?
Yes, both are enabled. It's not my gens or workflow so I'm not comfortable posting it. As mentioned, the total raw metadata comes to around 50k characters per image. It's a quite large comfyui workflow, and you need a number of unique images with the same size workflow to trigger the crash.
See if this improves things
https://github.com/RupertAvery/DiffusionToolkit/actions/runs/14348000145/artifacts/2908080007