ctransformers
ctransformers copied to clipboard
marella
Python bindings for the Transformer models implemented in C/C++ using GGML library.
Hi! It would fantastic if it supported Llama3. I hit a bug, and I'm unsure whether or not it is caused by quantization or by Ctransformers not being able to...
I am using the base code as given in HF from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml') print(llm('AI is going to')) While this works fine locally on Mac machine, it...
I tried running the OpenHermes model on kali linux using ctransformers tokenizer = AutoTokenizer.from_pretrained("teknium/OpenHermes-2.5-Mistral-7B") model_path = "openhermes-2.5-mistral-7b.Q4_K_M.gguf" model = AutoModelForCausalLM.from_pretrained(model_path, gpu_layers=16, threads=1, context_length=4096, max_new_tokens=-1) and i always get this error:...
You can modify the location where the Hugging Face model should be stored (when using the `transformers` library) by setting the environment variable `TRANSFORMERS_CACHE`. The default location is the directory...
Hi, I use debian 11, python 3.10, anaconda3 langchain==0.0.240 RAM = 16 Go swap = 10 Go SSD = 60 GO I try to use the simplest program with Ctransfors:...
I'm using `cTransformers` with `streamlit` to provide a UI to an LLM. After running the model once on my machine and closing the application and trying to start up again,...
**Problem:** Unable to install the package on a Linux machine with an AMD 6800XT GPU using ROCm. ``` docker run -it --device=/dev/kfd --device=/dev/dri --group-add video docker.io/rocm/dev-ubuntu-22.04:5.6-complete bash CT_HIPBLAS=1 pip install...
Using the command `$ CC="/opt/rocm/llvm/bin/clang" CXX="/opt/rocm/llvm/bin/clang++" CT_HIPBLAS=1 pip install ctransformers --no-binary ctransformers` I am unable to compile ctransformers for ROCM. I've tried with ROCM 5.5.3 as well as 5.6.1, and...
I have a use case where I need to modify input embeddings before they ``` from ctransformers import AutoModelForCausalLM llm = AutoModelForCausalLM.from_pretrained("TheBloke/Llama-2-7b-Chat-GGUF", model_file="llama-2-7b-chat.Q4_K_M.gguf", model_type="llama", gpu_layers=50) embeddings = llm.embed('Some text') #...
