ctransformers
ctransformers copied to clipboard
marella
Python bindings for the Transformer models implemented in C/C++ using GGML library.
Hi @marella Hope all is OK with you. I'm getting various reports on my repos with issues with CTransformers, and I see it's not been updated for nearly 2 months...
**env:** transformers ==4.35.2 ctransformers==0.2.27+cu121 ``` from ctransformers import AutoModelForCausalLM, AutoTokenizer model_name = "/home/me/project/search_engine/text-generation-webui/models/OpenHermes-2.5-Mistral-7B-GGUF/openhermes-2.5-mistral-7b.Q5_K_M.gguf" def load_model(model_name: str): model = AutoModelForCausalLM.from_pretrained(model_name, hf=True) tokenizer = AutoTokenizer.from_pretrained(model) return model, tokenizer tokenizer, model = load_model(model_name)...
~~~~~~~~~~~~~~~~~~~~~~~ Any use of "gpu_layers" crashes it. ~~~~~~~~~~~~~~~~~~~~~~~ CUDA is working: (ct) C:\Users\Jeremy\Documents>python Python 3.11.6 (tags/v3.11.6:8b6ee5b, Oct 2 2023, 14:57:12) [MSC v.1935 64 bit (AMD64)] on win32 Type "help", "copyright",...
Hi, When the LLM generate a long answer that exceeded 512 tokens, the program start to show warning message like this: WARNING:ctransformers:Number of tokens (513) exceeded maximum context length (512)...
Does ctransformers support Sliding Window Attention (SWA) ?
Hi, I'm having trouble with Mistral because the model is not loading on GPU but it is only running on CPU. That's the code: ``` from ctransformers import AutoModelForCausalLM, Config,...
GGML_ASSERT: D:\a\ctransformers\ctransformers\models\ggml/llama.cpp:453: data I get this error sometimes when loading a model. At first, I thought it was a corrupted model, and I redownloaded it which fixed the issue. However,...
When running: ``` from ctransformers import AutoModelForCausalLM, AutoTokenizer model = AutoModelForCausalLM.from_pretrained("radames/phi-2-quantized", hf=True) ``` I get: `RuntimeError: Failed to create LLM 'phi-msft' from '/root/.cache/huggingface/hub/models--radames--phi-2-quantized/blobs/77971e348da4b424832a089f812ba50dec2bd633ae39d26b4f2c89c0ff3dea27'.` However, when I use transformers, it runs...
I wonder why Zephyr is not yet supported in ctransformers. ``` from ctransformers import AutoModelForCausalLM m = AutoModelForCausalLM.from_pretrained('TheBloke/stablelm-zephyr-3b-GGUF', model_file='stablelm-zephyr-3b.Q4_K_M.gguf') ``` The above code actually downloaded the file from huggingface repo,...
Hello, It doesnt look like it's possible to download models outside hf ecosystem? For ex, I have some models hosted on a local MinIO and I cannot feed `from_pretrained("http...` path....
