torchtitan icon indicating copy to clipboard operation
torchtitan copied to clipboard
verified publisher icon pytorch

Enable CP

Open fegin opened this issue 1 year ago • 0 comments

Stack from ghstack (oldest at bottom):

  • -> #433

This PR adds experimental flags and functions to enable context parallelism. We currently support on ly FSDP + CP and CP only. CP + TP is being tested.

fegin avatar Jun 27 '24 21:06 fegin