OpenLLM
OpenLLM copied to clipboard
bentoml
Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
### Describe the bug When running `openllm start opt` for the first time, the process fails after downloading `..._config.json`. I'm on a Macbook Pro M2. Here's the output: ``` (openllm)...
### Feature request I want to use OpenLLM with available models to run on Apple M1/M2 processors (GPU support) through MPS. Today: ``` openllm start falcon No GPU available, therefore...
### Describe the bug Hi there, I followed the instruction on GitHub to start tiiuae/falcon-7b. ` pip install "openllm[falcon]" openllm start falcon --model-id tiiuae/falcon-7b ` Then, when calling the localhost:3000...
Hello, Are you planning to add support for parameter efficient finetuning methods? Also does it support doing inference using those adapters models to optimize VRAM? Thanks
### Describe the bug Hello, I followed the instructions on github (nothing more) and when I try to run it with the following command: `sudo docker run -it --rm -p...
### Describe the bug I'm running through the most basic install. I have creates an empty virtualenv with python 3.11. I've run `pip install openllm`, and I get a crash...
### Feature request openllm.LLMConfig should be serializable as a dict ```python config = openllm.AutoConfig.for_model('dolly-v2') dict(config) # should be the same as config.model_dump(flatten=True) ``` This means within the `LLMConfig` class generation,...
### Describe the bug Running project via README results in issues finding the necessary model. The missing step is something like `openllm download dolly-v2` before `openllm start dolly-v2` can be...
### Feature request Being able to use GGML models using ctransformers https://github.com/marella/ctransformers or llama.cpp https://github.com/abetlen/llama-cpp-python ### Motivation CPU support for Starcoder and eventually Falcon models, and overall perf improvements. ###...
