janhq
idea: non-nvidia appimage/deb distribution
Problem Statement
Since 0.5.8 the size of appimage added more than 1GB, with most of it being nvidia/cuda libraries.
Feature Idea
I understand that in LLM sphere space and traffic conservation is not a primary concern and AMD cards are second class, but it would be great if there was either non-nvidia AppImage without added bloat or, like cortex, jan could download needed libraries on demand.
@SolitudeSF This is a very solid suggestion, we'll calculate the pros & cons of both approaches & let you know
FWIW, the llama.cpp Vulkan backend recently got a lot of improvements (https://github.com/ggml-org/llama.cpp/discussions/10879), and runs on pretty much any desktop GPU, with far smaller dependencies.
+1, there are use cases where only UI is necessary (laptop without GPU (ok, there is UHD Graphics 620) accessing OpenAI compatible API that is SSH port forwarded from the desktop PC running cortex.cpp / llama.cpp / llama-swap / any other compatible setup).
So maybe separate build without any server parts?
