torchtitan icon indicating copy to clipboard operation
torchtitan copied to clipboard

remove PP tracer

Open tianyu-l opened this issue 1 year ago • 0 comments

Stack from ghstack (oldest at bottom):

  • -> #555

Discussed with @wconstab and @kwen2501 , it seems PP tracer has two limitations right now:

  1. It doesn't support init_weights, thus requiring a seed checkpoint to do init, which is something we probably will deprecate soon after we have the init support for manual splitting.
  2. It doesn't support mixed precision training, as the tracer requires that the model being traced be consistent with the model during forward.

We came to the conclusion to remove PP tracer for now, to have a clean version of torchtitan.

tianyu-l avatar Aug 22 '24 00:08 tianyu-l