integrate offline LLM for searching media in gallery

[shamirahim111]

trafficstars

Checklist

• [x] I made sure that there are no existing issues - open or closed - to which I could contribute my information.
• [x] I have read the FAQ and my problem isn't listed.
• [x] I have taken the time to fill in all the required details. I understand that the feature request will be dismissed otherwise.
• [x] This issue contains only one feature request.
• [x I have read and understood the contribution guidelines.

Is your feature request related to a problem? Please describe. there is no "LLM" or the buzzword "AI" in gallery. While this might be difficult to do on its own, there is a real problem of finding photos of something i took last month or photos of a friend or something.

Describe the solution you'd like https://github.com/slavabarkov/tidy this app exists and i've been using it for quite some time and it is very nice. the UX is "not" up to the quality of gallery so i was hoping if tidy could be integrated in gallery itself so that you can use the existing search that you have and the app would filter results. That's all i am asking for.

Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.

Additional context Add any other context or screenshots about the feature request here.

[naveensingh]

We'll maybe do this eventually once there are more lightweight and efficient solutions available for this (I'm yet to do any actual research on this but a lighter solution shouldn't take long given the pace things are advancing).

I haven't done any work with Open AI clip specifically (used by tidy): The tidy APK is 157MB so I'm guessing the model that is being used for this is too big even after quantization. I think most people do not want to download a 150MB gallery even if storage and bandwidth are relatively cheap now a days.

We could try this as an extension app for Gallery if enough people want it.

[shamirahim111]

@naveensingh i understand the idea but i for one really "desire" this feature. how about a setting inside the app itself that downloads the model and does its thing?

[shamirahim111]

https://github.com/FossifyOrg/Gallery/discussions/20

have started a poll

[jherazob]

I started using Simple Gallery because it was A SIMPLE GALLERY APP, adding this kind of thing is blatantly unnecessary bloat, the example you mention is 157Mb! And there's no reducing this any further, it's a fundamentally heavy application completely unnecessary for the tasks of A SIMPLE GALLERY APP.

[test2a]

@jherazob i would say the gallery job itself is being done nicely so like @naveensingh is saying, maybe we can do this as an extension app outside of the normal app so that people who need the enhanced functionality can still continue to use the app

[johnnynoone]

@shamirahim111 I like the idea of optionally downloading the model in the settings. This way language-specific models can be offered in the future - not everyone speaks English.

[GfEW]

Agreed, if you decide to include such "useful bloat" (even if it's become lighter by then), please make it modular.

I'd like to add that "AI" search is not only heavier than, but also not generally superior to strict pattern matching. Just think of finding (or counting, examining, sharing etc.) all items that exactly match your carefully crafted search input, in any LLM driven search - you'll have a hard time.

Therefore, even when the user activates LLM search, strict pattern matching should remain available at hand, without jumping through hoops (like e. g. first having to dive into settings, find the appropriate option there, change it, leave settings, to finally being able to perform the actual search).

[inson1]

@johnnynoone you cant do that, the app would need internet access, it has to be another app

EDIT max link would be possible

[haldi4803]

Here is another example of a fullfetched Gally with included CLIP search. https://github.com/greyovo/PicQuery

I think that (or similar) needs to be included in any gallery in the "Future". The Question is would that be now, in 2 years or in 5 years.

[inson1]

yes, but the app is huge 175 mb apk, how much would be the app? 350-400 mb? that has to be different and optional app

or wait for smaller size

[haldi4803]

or... sepparate loading of AI model.

[inson1]

@haldi4803 from where? No internet access

[haldi4803]

The Manual way... download the CLIP file you want from huggingface, and then select it in the app from the download location. But that would be more than just "simple"

[inson1]

@haldi4803 yea I guess the easiest way is to wait (as there is lot of work elsewhere anyway) or second app with it

[johnnynoone]

@johnnynoone you cant do that, the app would need internet access, it has to be another app

Oh, right. Then perhaps an extension installed separately from F-Droid? Just like the "OsmAnd Contour lines" extension for OsmAnd? Then it could be used in multiple Fossify apps, e.g. in the file manager.

[Panak252]

I'm not an android app developer so I wouldn't know if what I'm about to recommend makes any sense, but here's my idea:

A fork/variant of the gallery app could be made with modularity in mind, that allows for adding modules like image analysis (for objects, colours, whatever) and others, which can be "plugged in" (think of HDMI port connector covers) in such a way that allows for it to function both without any modules, as well as with any modules that come to mind.

This is just a thought, should something like this be considered desirable by both the developers and the users. This idea probably has flaws, but I believe you get the gist of it ;)