dynamic-batching topic
List
dynamic-batching repositories
InsNet
65
Stars
12
Forks
Watchers
InsNet Runs Instance-dependent Neural Networks with Padding-free Dynamic Batching.
batch-inference
65
Stars
0
Forks
Watchers
Dynamic batching library for Deep Learning inference. Tutorials for LLM, GPT scenarios.
grps
147
Stars
13
Forks
Watchers
【深度学习模型部署框架】支持tf/torch/trt/trtllm/vllm以及更多nn框架，支持dynamic batching、streaming模式，支持python/c++双语言，可限制，可拓展，高性能。帮助用户快速地将模型部署到线上，并通过http/rpc接...