Comparison with deepspeed inference?

[allanj]

as title mentioned

[shihaobai]

Thank you for your attention. As deepspeed primarily focuses on the static inference performance of models, while we are more concerned with the throughput of the entire inference service, we haven't compared it directly. We would greatly appreciate it if you could provide some comparative results.