FlagEmbedding icon indicating copy to clipboard operation
FlagEmbedding copied to clipboard

Published 20 hours ago • verified publisher icon FlagOpen

deepspeed训练m3模型，OOM

Open JOHNNY-fans opened this issue 10 months ago • 5 comments

四卡40gH卡，batch开32，开了gradient_checkpointing，query_max_len 512 ，passage_max_len 8192 训着训着会突然OOM，求问稳定的解决方案，（尝试了batch开4，16，32都会炸）

JOHNNY-fans avatar Apr 16 '24 05:04 JOHNNY-fans