ChatGLM-6B icon indicating copy to clipboard operation
ChatGLM-6B copied to clipboard
verified publisher icon THUDM

[Feature] <如何基于deepspeed实现单机多卡或者多机多卡的推理？>

Open clintlong opened this issue 2 years ago • 2 comments

Is your feature request related to a problem? Please describe.

No response

Solutions

有人实现了基于deepspeed框架进行多机多卡推理吗？

Additional context

No response

clintlong avatar Apr 19 '23 12:04 clintlong

一个gpu起一个Server，最后用nginx负载。

wfuqiang1982 avatar Apr 19 '23 17:04 wfuqiang1982

https://github.com/microsoft/Megatron-DeepSpeed 这个项目可以帮助deepspeed实现多卡多机推理，但需要改动模型

sharlec avatar Apr 28 '23 16:04 sharlec