ColossalAI
ColossalAI copied to clipboard
[FEATURE]: accelerate auto-parallel intra-op solver solving time
Describe the feature
The solving time for auto-parallel intra-op solver is unacceptable for LLMs as the number of layers increasing.
We could do following steps to reduce the solving time for LLMs:
- find largest repeated blocks.
- Use an alias set to force all repeated blocks sharing a same distributed training strategies.