ring-flash-attention icon indicating copy to clipboard operation
ring-flash-attention copied to clipboard

Published 20 hours ago • verified publisher icon zhuzilin

多机训练速度问题

Open kakaxi-liu opened this issue 8 months ago • 3 comments

我尝试在多机多卡上训练，发现耗时相比单机上要增加很多，想同训练环境下相比Deepspeed Ulysses耗时增加了三倍，而单机上却没有这个问题，请问是什么原因导致的呢？

kakaxi-liu avatar Jun 18 '24 06:06 kakaxi-liu