torchrec icon indicating copy to clipboard operation
torchrec copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

question about parallelism for embedding

Open imh966 opened this issue 1 year ago • 10 comments

It seems torchrec does not support the combination of data parallelism and row-wise parallelism for embedding. I want to know is there a plan on it? Or is row-wise parallelism efficient enough when it comes to multi-node training?

imh966 avatar Jun 17 '24 02:06 imh966