paxml icon indicating copy to clipboard operation
paxml copied to clipboard

Published 20 hours ago • verified publisher icon google

[Feature Request] Need ZeRo-1/2 to cooperate with PP+TP+DP. Which may more faster than FSDP sometimes.

Open MoFHeka opened this issue 5 months ago • 0 comments

For example in single A100 machine. Llama2 13B training speed with TP2 DP 4 + Zero1 is more faster than FSDP.

MoFHeka avatar Jan 26 '24 10:01 MoFHeka