ctransformers
ctransformers copied to clipboard
marella
Python bindings for the Transformer models implemented in C/C++ using GGML library.
Huggingface transformers offers `stopping_criteria`: https://huggingface.co/transformers/v4.6.0/_modules/transformers/generation_stopping_criteria.html I use this with a `threading.Event` so I can stop generation from a separate thread, and it works great with `transformers`: ```python def custom_stopping_criteria(local_llm_stop_event): def...
I'm working on a Mac with an M2 chip. I've installed ctransformers with metal support, and am setting up the model like below. However, when I check what device the...
### Description: Recent usage of the `pip` command with the `--no-binary` option has led to a deprecation warning indicating upcoming changes in `pip` 23.1. The `--no-binary` option, which has so...
I just used base script from the TheBloke repo and got an error during loading the model: OSError: libcudart.so.12: cannot open shared object file: No such file or directory my...
transformers version: `pip install transformers==4.34.0` ctransformersversion: `pip install ctransformers==0.2.27` I encounter the following error ``` File ".venv\lib\site-packages\ctransformers\transformers.py", line 84, in __init__kages\ctransformers\transformers.py", line 84, in __init__ super().__init__(**kwargs) File ".venv\lib\site-packages\transformers\tokenization_utils.py", line 366,...
When i run ctransformers[cuda], i get the error: CUDA error 35 at /home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu:4236: CUDA driver version is insufficient for CUDA runtime version However, the path "/home/runner/work/ctransformers/ctransformers/models/ggml/ggml-cuda.cu" does not exit. And...
There is a `rwkv.cpp` whch is also powered by ggml, So I think this is possible
Thanks for creating this library! We are using ctransformers at [PostgresML](https://postgresml.org/) to support GGUF models from Huggingface. We need to detect and recover from CUDA out of memory errors. Currently,...
I have installed ctransformers using - `pip install ctransformers[cuda]` I am trying following piece of code - ``` from langchain.llms import CTransformers config = {'max_new_tokens': 512, 'repetition_penalty': 1.1, 'context_length': 8000,...
