ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

[FEATURE]: accelerate auto-parallel intra-op solver solving time

Open YuliangLiu0306 opened this issue 2 years ago • 0 comments

Describe the feature

The solving time for auto-parallel intra-op solver is unacceptable for LLMs as the number of layers increasing.

We could do following steps to reduce the solving time for LLMs:

  1. find largest repeated blocks.
  2. Use an alias set to force all repeated blocks sharing a same distributed training strategies.

YuliangLiu0306 avatar Feb 21 '23 09:02 YuliangLiu0306