llm-inference topic

Run any open-source LLMs, such as Llama 3.1, Gemma, as OpenAI compatible API endpoint in the cloud.

bentoml

A programming framework for agentic AI 🤖 (PyPi: autogen-agentchat)

microsoft

Official inference library for Mistral models

mistralai

LLM.swift is a simple and readable library that allows you to interact with large language models locally with ease for macOS, iOS, watchOS, tvOS, and visionOS.

eastriverlee

[HPCA'21] SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning

mit-han-lab

export llama to onnx

luchangli03

Friendli: the fastest serving engine for generative AI

friendliai

An endpoint server for efficiently serving quantized open-source LLMs for code.

wangcx18

Tree prompting: easy-to-use scikit-learn interface for improved prompting.

csinva

Language models in C#

K024