PiPPy
PiPPy copied to clipboard
Decouple graph interpretation from pipeline executor
Graph interpretation refers to:
- Figuring out stage module to rank mapping
- Figuring out stage-to-stage communication relationship (connection, tensor transmission size, etc)
Pipeline executor refers to:
- Running micro-chunked data through pipeline stages based on certain schedule (FillDrain, 1F1B, etc)
Today, both implementations are in class PipelineStage. As work started to create a general pipeline executor that takes a manually cut model chunk, we should decouple the graph interpretation from the pipeline executor in PipelineStage. The general direction is to extract the stage-to-stage communication relationship and pass that to the new pipeline executor.
Graph interpretation: would help understand concepts such as GraphModule, Node.
Pipeline execution: would help understand P2P operations and how distributed program works.