LLaMA-Factory icon indicating copy to clipboard operation
LLaMA-Factory copied to clipboard

Published 20 hours ago • verified publisher icon hiyouga

关于单机多卡训练问题

Open Jingsong-Yan opened this issue 1 year ago • 2 comments

您好，请问如何实现将大模型的参数划分到多张卡上训练，而不是在每张卡上都加载整个模型参数。

Jingsong-Yan avatar Jun 17 '23 03:06 Jingsong-Yan

同问

taofennanhai avatar Jun 19 '23 08:06 taofennanhai

用 deepspeed zero3 试试

hiyouga avatar Jun 20 '23 02:06 hiyouga