tensorrt-llm topic
Awesome-LLM-Inference
📖A curated list of Awesome LLM Inference Paper with codes, TensorRT-LLM, vLLM, streaming-llm, AWQ, SmoothQuant, WINT8/4, Continuous Batching, FlashAttention, PagedAttention etc.
optimum-benchmark
🏋️ A unified multi-backend utility for benchmarking Transformers, Timm, PEFT, Diffusers and Sentence-Transformers with full support of Optimum's hardware optimizations & quantization schemes.
WhisperLive
A nearly-live implementation of OpenAI's Whisper.
cortex.cpp
Run and customize Local LLMs.
WhisperS2T
An Optimized Speech-to-Text Pipeline for the Whisper Model Supporting Multiple Inference Engine
openai_trtllm
OpenAI compatible API for TensorRT LLM triton backend
End-to-End-LLM
This repository is an AI Bootcamp material that consist of a workflow for LLM
Chat-With-RTX-python-api
Chat With RTX Python API
grps
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接...