ColossalAI
ColossalAI copied to clipboard
[FEATURE]: build alias set for repeat blocks and use it to reduce solving time
Describe the feature
The solving time for auto-parallel intra-op solver is unacceptable for LLMs as the number of layers increasing.
We could do following steps to reduce the solving time for LLMs:
-
find largest repeated blocks. (already done)
-
Use an alias set to force all repeated blocks sharing a same distributed training strategies. When we find a node belong to alias sets and it is not the first element of the set, we do not create a new lpVariable for it, but use the same lpVariable created for the first element of alias set.