PiPPy icon indicating copy to clipboard operation
PiPPy copied to clipboard

Decouple graph interpretation from pipeline executor

Open kwen2501 opened this issue 2 years ago • 0 comments

Graph interpretation refers to:

  • Figuring out stage module to rank mapping
  • Figuring out stage-to-stage communication relationship (connection, tensor transmission size, etc)

Pipeline executor refers to:

  • Running micro-chunked data through pipeline stages based on certain schedule (FillDrain, 1F1B, etc)

Today, both implementations are in class PipelineStage. As work started to create a general pipeline executor that takes a manually cut model chunk, we should decouple the graph interpretation from the pipeline executor in PipelineStage. The general direction is to extract the stage-to-stage communication relationship and pass that to the new pipeline executor.

Graph interpretation: would help understand concepts such as GraphModule, Node.

Pipeline execution: would help understand P2P operations and how distributed program works.

kwen2501 avatar Nov 22 '23 19:11 kwen2501