llama-cpp-python
llama-cpp-python copied to clipboard
abetlen
Python bindings for llama.cpp
# Prerequisites Please answer the following questions for yourself before submitting an issue. - [x] I am running the latest code. Development is very rapid so there are no tagged...
When will Qwen3-VL series support be added to llama-cpp-python? Is llama-cpp-python still actively maintained? I noticed the last commit was two months ago — I’m concerned about that too.
CUDA wheel installs, but GPU is never used on Windows 11 (Python 3.11, CUDA 12.1, torch finds GPU)
On Windows 11, with Python 3.11 and a CUDA 12.1-compatible NVIDIA GPU, I can successfully install llama-cpp-python via pip from the cu121 wheel, but no matter what, all model layers...
``` export PATH=/usr/local/cuda-11.4/bin:$PATH export LD_LIBRARY_PATH=/usr/local/cuda-11.4/lib64:$LD_LIBRARY_PATH NVCC_PREPEND_FLAGS="-allow-unsupported-compiler" CMAKE_ARGS="-DGGML_CUDA=on -DLLAVA_BUILD=off -DCMAKE_CUDA_FLAGS=-allow-unsupported-compiler" proxychains ../../bin/pip install -U llama-cpp-python ProxyChains-3.1 (http://proxychains.sf.net) |DNS-request| pypi.org |S-chain|--127.0.0.1:4445--4.2.2.2:53--OK |DNS-response| pypi.org is 151.101.0.223 |S-chain|--127.0.0.1:4445--151.101.0.223:443--OK Collecting llama-cpp-python Using cached llama_cpp_python-0.3.16.tar.gz (50.7...
I just tried compiling llama-cpp-python with `GGML_BACKEND_DL=ON` and `GGML_CPU_ALL_VARIANTS=ON` to make use of this nice feature with dynamic dispatch to a dynamically loaded backend, which e.g. made it possible to...
``` PS C:\Windows\system32> $env:CMAKE_GENERATOR = "MinGW Makefiles" PS C:\Windows\system32> $env:CMAKE_ARGS = "-DGGML_OPENBLAS=on -DCMAKE_C_COMPILER=C:/w64devkit/bin/gcc.exe -DCMAKE_CXX_COMPILER=C:/w64devkit/bin/g++.exe" PS C:\Windows\system32> pip install llama-cpp-python -i https://pypi.tuna.tsinghua.edu.cn/simple Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple Collecting llama-cpp-python Using cached https://pypi.tuna.tsinghua.edu.cn/packages/de/6d/4a20e676bdf7d9d3523be3a081bf327af958f9bdfe2a564f5cf485faeaec/llama_cpp_python-0.3.9.tar.gz...
``` # takes build time + 5-8 minutes to complete FROM nvidia/cuda:12.4.1-cudnn-devel-ubuntu22.04 ENV DEBIAN_FRONTEND=noninteractive ENV HF_TOKEN=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 ENV TZ=Asia/Hong_Kong # install linux packages RUN apt-get update && \ apt-get update &&...
I needed to add this DLL for the library to work on my system (possibly to do with a new cuda toolkit version, unclear). Possibly related to https://github.com/abetlen/llama-cpp-python/issues/2070. If anyone...
llama.cpp cli supports it but llama-cpp-python don't and we need it
