ava icon indicating copy to clipboard operation
ava copied to clipboard

Support Nvidia, AMD, AVX, ... (CUDA, ROCm, Vulkan and other BLAS) for windows

Open cztomsik opened this issue 9 months ago • 1 comments

This will take some time, so this is just rough sketch for later:

  • windows build/binary could expect llama.dll
    • could be straightforward, because zig can cross-compile so I can do this locally on my machine
  • zig build can download/extract a zip file from url like https://github.com/ggerganov/llama.cpp/releases/download/{short_rev}/llama-{short_rev}-bin-{blas}.zip
    • where short_rev is obtained from the llama.cpp git submodule
    • blas is something like win-cuda-cu11.7.1-x64 passed as -Dblas=xxx to a zig build
  • both .exe and .dll should be marked as artifact

After this is done, we can make a windows pipeline, with matrix for each BLAS, and hopefully, we will get a .zip file, which people can just download and run. Of course, they still need to have given BLAS installed on their system.

cztomsik avatar Apr 26 '24 07:04 cztomsik