llm-serving topic

Ray is a unified framework for scaling AI and Python applications. Ray consists of a core distributed runtime and a set of AI Libraries for accelerating ML workloads.

ray-project

A high-performance ML model serving framework, offers dynamic batching and CPU/GPU pipelines to fully exploit your compute machine

mosecorg

SkyPilot: Run AI and batch jobs on any infra (Kubernetes or 12+ clouds). Get unified execution, cost savings, and high GPU availability via a simple interface.

skypilot-org

RayLLM - LLMs on Ray

ray-project

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

bentoml

A high-throughput and memory-efficient inference and serving engine for LLMs

vllm-project

npm like package ecosystem for Prompts 🤖

sugarcane-ai

Superduper: Integrate AI models and machine learning workflows with your database to implement custom AI applications, without moving your data. Including streaming inference, scalable model hosting,...

superduper-io

🪶 Lightweight OpenAI drop-in replacement for Kubernetes

chenhunghan

Friendli: the fastest serving engine for generative AI

friendliai