lm-inference-engines icon indicating copy to clipboard operation
lm-inference-engines copied to clipboard
verified publisher icon lapp0

Comparison for other frameworks as well

Open ogencoglu opened this issue 2 years ago • 2 comments

ogencoglu avatar Dec 11 '23 06:12 ogencoglu

Not familiar with any of these other than FlexFlow unfortunately. Happy to include PRs for any of these if they are uniquely valuable inference engines.

lapp0 avatar Dec 11 '23 09:12 lapp0

Each has its own focus. Some benchmarks can be found on their docs e.g., https://github.com/InternLM/lmdeploy?tab=readme-ov-file#performance

ogencoglu avatar Dec 11 '23 11:12 ogencoglu