all-in-one icon indicating copy to clipboard operation
all-in-one copied to clipboard
verified publisher icon nextcloud

Add local-ai community container with Vulkan support

Open docjyJ opened this issue 1 year ago • 16 comments

A lot of GPU supported: https://vulkan.gpuinfo.org/listdevices.php?platform=linux

Image: https://github.com/docjyJ/aio-local-ai-vulkan

docjyJ avatar Dec 28 '24 15:12 docjyJ

@docjyJ thanks a lot for this PR! :)

szaimen avatar Jan 06 '25 09:01 szaimen

Btw @docjyJ have you had the chance to look a bit further into this? :)

szaimen avatar Jun 18 '25 08:06 szaimen

Not yet, it's quite complicated to have Vulkan on my server. I have a very old version of the Linux kernel so I need to find a solution to update.

docjyJ avatar Jun 30 '25 20:06 docjyJ

Local AI doesn't have a Vulkan image for arm64... Since my cloud is on arm, I'm stuck...

I've opened an issue: https://github.com/mudler/LocalAI/issues/5778

docjyJ avatar Jul 02 '25 00:07 docjyJ

I have Vulkan on my PC and am running a local instance of Nextcloud over HTTP, so I opted for simplicity with automatic configuration and explanations for accessing the web interface.

docjyJ avatar Jul 02 '25 11:07 docjyJ

Thanks @docjyJ for continuing the work on this! 😊

I mean you could also try to build the image from source instead of proxying it. Then you could also build for arm64... However of course not sure how feasible this is...

szaimen avatar Jul 02 '25 12:07 szaimen

image It's using my GPU. But I don't know why it's so slow...

docjyJ avatar Jul 02 '25 12:07 docjyJ

So WDYT ?

docjyJ avatar Jul 02 '25 13:07 docjyJ

See that https://www.reddit.com/r/LocalLLaMA/comments/1j1swtj/vulkan_is_getting_really_close_now_lets_ditch/

docjyJ avatar Jul 02 '25 18:07 docjyJ

So WDYT ?

I was wondering if it would be possible to make this container the default LocalAI container in AIO instead of being a variant?

Of course, we would need to resolve the arm64 problem first.

Would it be possible to simply build the project from sources in the Dockerfile? WDYT?

szaimen avatar Jul 03 '25 08:07 szaimen

Yes, I think it could be the main container.

ARM support has an open PR: https://github.com/mudler/LocalAI/pull/5780

docjyJ avatar Jul 03 '25 10:07 docjyJ

Yes, I think it could be the main container.

Cool!

ARM support has an open PR: mudler/LocalAI#5780

I see... Honestly, I would like to wait a few weeks and see if arm64 support evolves upstream so that we can push this forward here...

szaimen avatar Jul 04 '25 06:07 szaimen

yes !

docjyJ avatar Jul 04 '25 17:07 docjyJ

I added web ui access with basic auth, we can support caddy community container. Updated doc and use local AI image without preconfigured models (to save space disk, user can install models from local AI web ui)

docjyJ avatar Jul 06 '25 11:07 docjyJ

FYI: you can now use string replacement in nextcloud_exec_commands since https://github.com/nextcloud/all-in-one/pull/6835. See https://github.com/nextcloud/all-in-one/pull/6835 for example.

szaimen avatar Sep 08 '25 09:09 szaimen

I saw that PR has had the Roadmap label for two weeks. Waiting for news.

docjyJ avatar Sep 27 '25 10:09 docjyJ