Making the app faster for users with massive libraries (7 million indexed images)

[kardo22]

Is your feature request related to a problem? Please describe. Im merely using this app to display images and i wanted it to be fast, fully indexed my library has nearly 7 million images, it is far from fast. It seems even though i have disabled everything from face detection, searching, listing of how many files are in folders etc there are still a ton of heavy queries running when opening a directory of files.

Describe the solution you'd like An option in the settings panel that disables all the advanced features and merely just uses the index to show albums without having to actually scan the disk for location.

Describe alternatives you've considered (optional) Im not a coder but i believe just having a couple of different queries that don't do all these updating and sorting based on metadata could make the app a lot faster for us with massive libraris (mysql is a must ofcourse, this option should only be available if you have mysql enabled).

[bpatrik]

Can you describe you use-case more in depth?

Also how many photos do you have in a single folder? Having less than 1k photos per folder, should be still fast. The app slows down if you have significantly more. See #437 for listing too many photos.

[fengwang]

I encountered this problem as well. I have a single folder containing about 3 million images, which is a remote dataset for my deep learning applications. I would like to use pigallery as a default viewer to cherry pick images of special interest, but the seed is too slow.

[bpatrik]

I'm sorry but the app is not made for such use-cases. :/ It's a family photo gallery.

I don't think I would have a time in the near future O(years) to make it usable.

-- Sorry for being brief, sent from my phone.

On Sun, 26 Jun 2022, 23:52 feng wang, @.***> wrote:

I encountered this problem as well. I have a single folder containing about 3 million images, which is a remote dataset for my deep learning applications. I would like to use pigallery as a default viewer to cherry pick images of special interest, but the seed is too slow.

— Reply to this email directly, view it on GitHub https://github.com/bpatrik/pigallery2/issues/496#issuecomment-1166655352, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABZKA5QFMJCGGX4NKKDT3SLVRDGLJANCNFSM5X3JTL3Q . You are receiving this because you commented.Message ID: @.***>

[wonx]

Hi, I have also encountered the problem that some search results have too many pictures and pigallery2 keeps "loading" indefinitely without showing any pictures. With up to 8000-10000 pictures, it seems to work fine, but more than that it might fail to load.

What about a workaround, and handle a maximum number of pictures per search / album, so at least the first x000 are shown? This is something that could be customized as a value in the settings.

[wonx]

(nevermind, I saw that option already exists in the settings)

[bpatrik]

I made an update in the README that app is made to support O(100k) photos having <2k photos per folder. I made significant performance improvements lately, but 7 mill is def too much to handle.

So I will just close this as infeasable