ColossalAI icon indicating copy to clipboard operation
ColossalAI copied to clipboard

[FEATURE]: build alias set for repeat blocks and use it to reduce solving time

Open YuliangLiu0306 opened this issue 2 years ago • 0 comments

Describe the feature

The solving time for auto-parallel intra-op solver is unacceptable for LLMs as the number of layers increasing.

We could do following steps to reduce the solving time for LLMs:

  • find largest repeated blocks. (already done)

  • Use an alias set to force all repeated blocks sharing a same distributed training strategies. When we find a node belong to alias sets and it is not the first element of the set, we do not create a new lpVariable for it, but use the same lpVariable created for the first element of alias set.

YuliangLiu0306 avatar Feb 27 '23 04:02 YuliangLiu0306