llm-rs-python

llm-rs-python copied to clipboard

Will you please guide how to run the conversion script?

1

Hie 👋🏻 Coming from [this](https://github.com/ggerganov/ggml/blob/master/examples/gpt-j/convert-h5-to-ggml.py) GGML conversion script and the issue that you commented in https://github.com/ggerganov/ggml/issues/280#issuecomment-1606901828 ## Now... I have a fine-tuned model found on huggingface called "**NumbersStation/nsql-350M**" which basically...

AayushSameerShah

thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidMagic { path: "model_merak.bin" }', src/model.rs:47:12

3

Hi Author, I tried to implement llm-rs in my jupyter. After i installed RUST the error still occured : thread '' panicked at 'called `Result::unwrap()` on an `Err` value: InvalidMagic...

widyaputeriaulia10

How much RAM needed to convert gpt2 13b model to ggml using your Manual convert function?

1

I'm trying to convert it on 16gb RAM but converting process seems to last forever.

JohnClaw

Add llm-rs-python to haystack-integrations

7

Hey @LLukas22! I'm Stefano and I work on the **Haystack** project. I found `llm-rs-python` by chance and started experimenting with it on my laptop and was really impressed! 💣 So...

anakin87

Is streaming supported with langchain AsyncIteratorCallbackHandler?

1

I am getting no reponses when using with langchain callback AsyncIteratorCallbackHandler? It gives only this warning ``` RuntimeWarning: coroutine 'AsyncCallbackManagerForLLMRun.on_llm_new_token' was never awaited run_manager.on_llm_new_token(chunk, verbose=self.verbose) RuntimeWarning: Enable tracemalloc to get...

AdrianLsk

GPU Not Utilized When Using llm-rs with CUDA Version

2

I have installed the llm-rs library with the CUDA version, However, even though I have set `use_gpu=True` in the `SessionConfig`, the GPU is not utilized when running the code. Instead,...

andri-jpg

Feature Request: Falcon 7B support

1

I tried MPT end to end in script form. Works like charm. Can you provide a similar support to new Falcon model?

sidharthiimc