Ravindra Marella

You can get the logits using [`llm.logits`](https://github.com/marella/ctransformers#property-llmlogits) property. In order to implement `forward`, you can use the low-level [`llm.eval()`](https://github.com/marella/ctransformers#method-llmeval) method. I have done some work on this in the past...

Looks like outlines `generate` needs a custom model and tokenizer object which can be created using `Transformers` and `TransformersTokenizer` classes. I recommend using the original HF tokenizer to simplify things:...

Please post the full error message with stack trace and make sure you are using the latest version: ```sh pip install ctransformers --upgrade ```

Yes, custom/HF tokenizer can be used with the [`generate()`](https://github.com/marella/ctransformers#method-llmgenerate) method: ```py from ctransformers import AutoModelForCausalLM from transformers import AutoTokenizer tokenizer = AutoTokenizer.from_pretrained('gpt2') llm = AutoModelForCausalLM.from_pretrained('marella/gpt-2-ggml') tokens = tokenizer.encode('AI is going...

I haven't used the guidance library but the `guidance.llms.transformers.LLaMA` class is expecting HF transformers object but you are passing ctransformers object, so it won't work. It looks like there is...

I would also like to add support for it but it doesn't seem to have documentation on how to add new models. I will try to follow this [example](https://github.com/microsoft/guidance/issues/58#issuecomment-1555348871) and...

Hey, I implemented a 🤗 Transformers compatible model and tokenizer using ctransformers and was able to run one of the examples but I think it has some bugs. I will...

Hi, I pushed the changes to `guidance` branch. You can install using: ```sh git clone https://github.com/marella/ctransformers cd ctransformers git checkout guidance pip install -e . ``` and use it as:...

Hi, such cases are already [handled](https://github.com/marella/ctransformers/blob/744c5e29dde6d82d7e0351e0bc7a9dddbfb553bb/ctransformers/llm.py#L531-L534), so it shouldn't happen. Can you please share the code, prompt and link to model you are using.

Hi, it looks like a memory issue. How much RAM do you have?