cookbook icon indicating copy to clipboard operation
cookbook copied to clipboard

Add communication volume calculation script

Open Quentin-Anthony opened this issue 1 year ago • 0 comments

Would be good to model the communication volume in bytes of a given parallelism setup. Situations to model:

  • Different parallelism schemes
    • ZeRO-1/2/3, ZeRO++
    • 3D parallelism
  • Activation checkpointing
  • Different dtypes

Bonus points:

  • % volume breakdown separated by collective

Quentin-Anthony avatar Jan 04 '24 15:01 Quentin-Anthony