ctransformers
ctransformers copied to clipboard
marella
Python bindings for the Transformer models implemented in C/C++ using GGML library.
What cuda drivers are needed? I'm able to run llama-cpp-pyhon and llama.cpp just fine, but not this tool Ubuntu 22 Cuda 11.7 driver is 515.43.04
Hi, I am trying to use "**TheBloke/WizardCoder-Guanaco-15B-V1.0-GGML**", however, I am getting following error: ``` GGML_ASSERT: /home/runner/work/ctransformers/ctransformers/models/ggml/ggml.c:4103: ctx->mem_buffer != NULL Aborted ``` I get same error with abacaj's replit inference code,...
Hi there, What's the process for adding new models? I'd theoretically like to quantize and run a version of Meta's [Nougat model](https://facebookresearch.github.io/nougat/) in CTransformers at some point. I'd be happy...
I started out with experimenting a bit with CTransformers. The device I have been using is ``` ASUS Laptop 16 GB RAM 6 GB NVIDIA RTX 3060 ``` And I...
Hello, This is my first time posting at any repository of GitHub so if i have made any mistake apologies in advance. 🙏 I am using "wizardcoder-python-7b-v1.0.Q4_K_M.gguf" model for generating...
I am trying to run the project on Linux and I installed the library using following: ```python CMAKE_ARGS="-DLLAMA_BLAS=ON -DLLAMA_BLAS_VENDOR=OpenBLAS" pip install ctransformers ``` ## Q: How to know if my...
Hi Team Is there any way to calculate the total number of tokens spends for the models which were consumed using ctransformers like how OPENAI is giving for their models...
this PR is for #154. I believe PreTrainedTokenizer `super().__init__(**kwargs)` attempted to use `get_vocab` from LLM but `self._llm = llm` is not yet set, hence it can not access to the...
# Overview I am using the `llama-7b-4_KM` model for my simple **Chat over docs** application which involves *fetching relevant chunks* and feeding (stuffing) them in the prompt. # Problem Each...
Is any update is going to come regarding the Starcoder support for as i can see the llama.cpp added StarCoder support on September 15th and Ctransformers hasn't been updated since...
