llama-cpp-python
llama-cpp-python copied to clipboard
abetlen
Python bindings for llama.cpp
I tried setting up `llama-cpp-python` in the current version `0.3.7` on my MacBook M4 Pro. In the first step I only installed via `pip install llama-cpp-python --no-cache-dir` without specifiying the...
**Issue:** I am trying to update llama-cpp-python to the latest version (0.3.7) to test the latest Distill version models of DeepSeek, such as DeepSeek r1 Distil Qwen. However, I am...
My graphics card was RTX3060 12G, the model used was Qwen2.5-7B-instruct-Q4_k_M, normally the model should only take up 4~5G VRam, so I thought the VRam of my GPU was sufficient...
Updates the requirements on [starlette-context](https://github.com/tomwojcik/starlette-context) to permit the latest version. Release notes Sourced from starlette-context's releases. v0.4.0 Drop support for Python 3.8. Now only 3.9+ is supported ( tomwojcik/starlette-context#152 )...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
Hello, I've been trying to build llama-cpp-python latest version (0.3.7)for a week for Deepseek R1. But i failed. Whenever i did succesfully it errors(access violation 0x000000) when import and run....
# Prerequisites Please answer the following questions for yourself before submitting an issue. - [X] I am running the latest code. Development is very rapid so there are no tagged...
I've just successfully installed it! Here's the information for your reference. PowerShell ： $env:CUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.6" $env:CMAKE_GENERATOR_PLATFORM="x64" $env:FORCE_CMAKE="1" $env:CMAKE_ARGS="-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89" pip install llama-cpp-python --no-cache-dir --force-reinstall --upgrade ********************************************************************** **...
For those trying to use GitHub Actions to build the latest version (v0.3.7) with CUDA 12.6 for Windows but failed, maybe installing VS version >=17.12 and CUDA directly like [here](https://github.com/Jimver/cuda-toolkit?tab=readme-ov-file#example-usage)...
(base) PS C:\WINDOWS\system32> conda activate CUDA124-py312 (CUDA124-py312) PS C:\WINDOWS\system32> $env:CUDA_TOOLKIT_ROOT_DIR="C:/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v12.4" (CUDA124-py312) PS C:\WINDOWS\system32> $env:CMAKE_GENERATOR_PLATFORM="x64" (CUDA124-py312) PS C:\WINDOWS\system32> $env:FORCE_CMAKE="1" (CUDA124-py312) PS C:\WINDOWS\system32> $env:CMAKE_ARGS="-DGGML_CUDA=ON -DCMAKE_CUDA_ARCHITECTURES=89" (CUDA124-py312) PS C:\WINDOWS\system32>...
**Is your feature request related to a problem? Please describe.** A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] **Describe the solution you'd...
