Liger-Kernel
Liger-Kernel copied to clipboard
Add GLM4_MOE model support
Summary
This PR adds support for GLM4.5 (GLM-4 MOE) models to the Liger Kernel #951 https://huggingface.co/zai-org/GLM-4.5 which share the same structure as GLM 4.6
Testing Done
For the convergence test on fp32, model size can easily leads to OOM, initially I was using 4090 to run the tests, however only fp32 encounters OOM, so I move forward to L40S to finish all the tests.
- Hardware Type: <L40S>
- [x] run
make testto ensure correctness - [x] run
make checkstyleto ensure code style - [ ] run
make test-convergenceto ensure convergence