torchtitan icon indicating copy to clipboard operation
torchtitan copied to clipboard

Published 20 hours ago • verified publisher icon pytorch

integrate with nccl-exp

Open wanchaol opened this issue 1 year ago • 1 comments

to enable things like zero-copy

wanchaol avatar Jan 29 '24 19:01 wanchaol

@wanchaol -- Looking at how XLformers does this, we need to build PT with thirdparty NCCL pointing to NCCL-exp. Does not look like any changes in torchtrain is needed.

gnadathur avatar Feb 27 '24 04:02 gnadathur