llm-serving topic

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-project

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

mosecorg

SkyPilot: Run LLMs, AI, and Batch jobs on any cloud. Get maximum savings, highest GPU availability, and managed execution—all with a simple interface.

skypilot-org

RayLLM - LLMs on Ray

ray-project

Run any open-source LLMs, such as Llama 2, Mistral, as OpenAI compatible API endpoint in the cloud.

bentoml

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project

npm like package ecosystem for Prompts 🤖

sugarcane-ai

🔮 SuperDuperDB: Bring AI to your database! Build, deploy and manage any AI application directly with your existing data infrastructure, without moving your data. Including streaming inference, scalab...

SuperDuperDB

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

chenhunghan

Friendli: the fastest serving engine for generative AI

friendliai