OpenLLM
OpenLLM copied to clipboard
bentoml
Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.
### Feature request The new version of Starcoder is out (about 4 months ago). Can you include it to the supported models. This comes when I try to start the...
Bumps [openai](https://github.com/openai/openai-python) from 1.47.0 to 1.50.2. Release notes Sourced from openai's releases. v1.50.2 1.50.2 (2024-09-27) Full Changelog: v1.50.1...v1.50.2 Bug Fixes audio: correct types for transcriptions / translations (#1755) (76c1f3f) v1.50.1...
![dependabot[bot] avatar](https://avatars.githubusercontent.com/in/29110?v=4 "dependabot[bot] avatar"){kind=avatar}
### Describe the bug I want to run Qwen0.5b on a k8s cluster without GPU, but the service startup has failed so far. Is there any way to support CPU...
### Describe the bug When I try to serve a llama 3.1 8B-4bit with openllm, it says that "This model's maximum context length is 2048 tokens". On https://huggingface.co/meta-llama/Meta-Llama-3.1-8B, it says...
### Describe the bug Hi! I tried to run an llm locally using `openllm`, and `phi3:3.8b-ggml-q4` happens to be the only model which I am able to run locally according...
updates: - [github.com/astral-sh/ruff-pre-commit: v0.12.12 → v0.14.4](https://github.com/astral-sh/ruff-pre-commit/compare/v0.12.12...v0.14.4) - [github.com/pre-commit/mirrors-mypy: v1.17.1 → v1.18.2](https://github.com/pre-commit/mirrors-mypy/compare/v1.17.1...v1.18.2) - [github.com/editorconfig-checker/editorconfig-checker.python: 3.4.0 → 3.4.1](https://github.com/editorconfig-checker/editorconfig-checker.python/compare/3.4.0...3.4.1)
![pre-commit-ci[bot] avatar](https://avatars.githubusercontent.com/in/68672?v=4 "pre-commit-ci[bot] avatar"){kind=avatar}
Bumps the actions-dependencies group with 3 updates in the / directory: [actions/upload-artifact](https://github.com/actions/upload-artifact), [actions/download-artifact](https://github.com/actions/download-artifact) and [softprops/action-gh-release](https://github.com/softprops/action-gh-release). Updates `actions/upload-artifact` from 4.6.2 to 5.0.0 Release notes Sourced from actions/upload-artifact's releases. v5.0.0 What's Changed...
Bumps the production-dependencies group with 2 updates in the / directory: [bentoml](https://github.com/bentoml/bentoml) and [openai](https://github.com/openai/openai-python). Updates `bentoml` from 1.4.23 to 1.4.24 Release notes Sourced from bentoml's releases. v1.4.24 What's Changed doc:...
### Feature request I am honestly new to this repo + relatively new to MCP. I have a general understanding of MCP client + MCP server I made a local...
### Describe the bug Found OpenLLM recently while researching abstractions for local model execution and abstraction. The [OpenLLM repo](https://github.com/bentoml/OpenLLM) stood out. I may have initially misunderstood the options for using...
