llama-cpp-python

llama-cpp-python copied to clipboard

Intel Mac (i9 - 5500M) (macOS 15.3.2) - ValueError: Failed to create llama_context - llama_init_from_model: failed to initialize Metal backend

1

# Prerequisites Please answer the following questions for yourself before submitting an issue. - [x] I am running the latest code. Development is very rapid so there are no tagged...

starkAhmed43

Publish the latest llama.cpp?

2

Hello, I run an AMD card and there have been very significant ROCm support updates (flash attention, quants, massive speed improvements) since the llama.cpp version currently in llama-cpp-python. Could you...

curvedinf

Add: Include option to normalize & truncate embeddings in create_embe…

2

## What - Includes option to create **normalized** embeddings with **truncation** while using the "create_embeddings" method. ## Why Currently, the "create_embeddings" methods does not include this as args. However, embed...

KanishkNavale

Add reranking support

10

According to the https://github.com/ggerganov/llama.cpp/pull/9510, lllama-cpp supported for reranking model https://huggingface.co/BAAI/bge-reranker-v2-m3. Please provide support for this version.

donguyen32

Stream last block not return token usage info when create_chat_completion_openai_v1 or create_chat_completion but server does

1

# Prerequisites Stream last block not return token usage info when create_chat_completion_openai_v1 or create_chat_completion but server does. Python api is better for me, but no token usage returned, is this...

hh23485

Include usage key in create_completion when streaming

2

**Is your feature request related to a problem? Please describe.** Since `create_completion` may yield text chunks comprised of multiple tokens per yield (e.g. in the case of multi-byte Unicode characters),...

zhudotexe

Feat: Support Ranking Method

4

# Description Hello @abetlen! Thank you for your work on this library. This PR introduces a new rank method in the Llama class, enabling users to rank documents based on...

yutyan0119

Prebuilt cuda wheel has a GLIBC version mismatch on Ubuntu 20.04.

2

I use pip to install this wheel successfully: https://github.com/abetlen/llama-cpp-python/releases/download/v0.2.90-cu124/llama_cpp_python-0.2.90-cp312-cp312-linux_x86_64.whl When running, it says `RuntimeError: Failed to load shared library '/usr/local/anaconda3/envs/py312cuda124/lib/python3.12/site-packages/llama_cpp/lib/libllama.so': /lib/x86_64-linux-gnu/libc.so.6: version 'GLIBC_2.32' not found (required by /usr/local/anaconda3/envs/py312cuda124/lib/python3.12/site-packages/llama_cpp/lib/libllama.so)` After searching,...

Moon-404

Hello. The rocm-hip-sdk on arch linux wants me to pull in 30gb of dependencies which is insane. I just want to compile llama-cpp-python (llama.cpp) with hip support, to see if...

madprops

Bumps [pypa/cibuildwheel](https://github.com/pypa/cibuildwheel) from 2.22.0 to 2.23.2. Release notes Sourced from pypa/cibuildwheel's releases. v2.23.2 🐛 Workaround an issue with pyodide builds when running cibuildwheel with a Python that was installed via...

dependabot[bot]