(FSDP or DDP) + PP support

[wconstab]

• [x] FSDP + PP
• [ ] DDP + PP
• [ ] DCP path

[gnadathur]

Includes validating DCP path

[23Uday]

Can PiPPY be used with DDP? As a PP model within a host and a DDP model inter hosts?

[kwen2501]

Yes it can. Here is an example: https://github.com/pytorch/pytorch/blob/3bc073d7280d36eec2af3b1969c165b336a8ecae/test/distributed/_composable/test_composability/test_pp_composability.py#L94

[23Uday]

Yes it can. Here is an example: https://github.com/pytorch/pytorch/blob/3bc073d7280d36eec2af3b1969c165b336a8ecae/test/distributed/_composable/test_composability/test_pp_composability.py#L94

Thank you for your response. Based on the code provided, it seems that I have to first split the model into various pipeline stages (PP) and wrap each stage in a DDP Module, then run a pipeline schedule, Is my understanding correct? Is the a documentation available which lays out the path for composability of various parallelism paradigms with PiPPy? Thank you.

[kwen2501]

Your understanding is correct. We are working on a 3D composability tutorial/documentation. Please stay tuned. Before that, you can refer to the parallelize_llama implementation in torchtitan: https://github.com/pytorch/torchtitan/blob/main/torchtitan/parallelisms/parallelize_llama.py

[kwen2501]

Tracking it here: https://github.com/pytorch/pytorch/issues/136408