Docker exiting without indication of why in logs

[whoopn]

How might I collected more detailed logs regarding system/docker util for Damselfly? Its exiting after about 12 hours of running.

Last line just mentions it read some metadata for an image, then it stops logging. It appears my unraid server is upset with the amount of RAM being consumed:

Dec 19 21:28:17 Tower kernel: Call Trace: Dec 19 21:28:17 Tower kernel: out_of_memory+0x3e8/0x41a Dec 19 21:28:17 Tower kernel: 0 pages HighMem/MovableOnly Dec 19 21:28:17 Tower kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0-1,global_oom,task_memcg=/docker/815bdb39890619b5a7e44e65de5b14d52be4f4d3b0ee345c839a9e66c6a92328,task=Damselfly.Web.S,pid=11603,uid=0 Dec 19 21:28:17 Tower kernel: Out of memory: Killed process 11603 (Damselfly.Web.S) total-vm:292710064kB, anon-rss:6920248kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:17248kB oom_score_adj:0

Question: What that Damselfly is doing would suddenly consume so much RAM unraid (or any other system) would automatically kill it? Can it be turned off?

My main goal with your app (and it does it REALLY well): Index all file names/folder and make them searchable I already upload my photos to google photos so I just need a file name and then I search and I find the full res versions of what I found on Google Photos. You've knocked it straight out of the park on the index/search, the fact that it shows the embedded previews and quickly shows thumbnails is nearly miraculous. While i would love to ditch google photos and I really want the AI/Close Match/Deduplicating capabilities, I'm willing to test them if thats helpful but I'd like to have one instance of Damselfly that is super simple: just index/search, I don't even care about metadata really. So is there a way I can turn off metadata and AI as well as thumbnail generation tasks?

[Webreaper]

There's an option to disable the AI stuff. I don't have an option to disable thumb generation, but am planning to add one, as it would be possible to generate them on the fly if your CPU is quick enough.

In terms of the RAM usage I'm not sure - I've not seen that....

[whoopn]

Ok I've disabled the AI stuff. I think I'd rather thumbnails but a suggestion if you find the time:

Allow different levels of thumbnail creation maybe? I'm fine with the embedded thumbnails/jpegs where applicable, but maybe don't bother making fullsize thumbnails until one clicks on the image? Or a setting to have them created on the fly?

I think with inflation folks are going to shoot for lower power usage overall so only creating thumbnails or finding faces on demand might be a sweet feature (and possibly something you could have a paid tier for if thats your aim ever).

[Webreaper]

Yep, that makes sense.

I had a plan to implement a couple of options:

1. Don't ever create large thumbs - just show the full-res image directly. For people who don't have 30MB images, or who have fast LANs, this might make sense and save CPU and disk space.

2. Ability to have all thumbs generated on the fly and not cached on disk (ie, save space at the cost of CPU power)

3. Always generate thumbs on the fly, but cache to disk so they're only generated once.

Hopefully that should fulfil your needs.

Unfortunately I'm trying to move house at the moment so probably won't get to this until end of Jan....

[whoopn]

Ok I have a follow up if its interesting to you:

I believe I know what is causing the out-of-memory issues, I ran Damselfly on my desktop (5950X/32GBRAM) and it crashed as well. I have a number of enormous panoramas (like 500MP-1.2GP). I'm thinking its hitting those and running out of runway.

Would it be difficult to implement a setting to not index files above a certain size? For me I'd set it to around 400MB.

[Webreaper]

Wow, that's a big image.

Your change sounds like a sensible one, I'll schedule it when I get time.