Pytorch profiler does not support Distributed view for FSDP training

Open HamidShojanazeri opened this issue 3 years ago • 1 comments

Running the FSDP training (released with PT 1.11) which is a type of DDP, it is expected to have distributed view in Pytorch profile. The distributed view is currently missing from profiler for FSDP.

Following are snapshots from FSDP vs DDP Profiler view.

FSDP training Screen Shot 2022-02-18 at 1 38 15 PM

DDP training.

Feature request

Adding support for FSDP in Distributed view of Profiler.

cc @zhaojuanmao , @mrshenli

Mar 17 '22 22:03 HamidShojanazeri

@rohan-varma do you know who can help with this?

Mar 22 '22 18:03 zhaojuanmao