Liger-Kernel icon indicating copy to clipboard operation
Liger-Kernel copied to clipboard

Add GLM4_MOE model support

Open vvvdwbvvv opened this issue 2 weeks ago • 1 comments

Summary

This PR adds support for GLM4.5 (GLM-4 MOE) models to the Liger Kernel #951 https://huggingface.co/zai-org/GLM-4.5 which share the same structure as GLM 4.6

Testing Done

For the convergence test on fp32, model size can easily leads to OOM, initially I was using 4090 to run the tests, however only fp32 encounters OOM, so I move forward to L40S to finish all the tests.

  • Hardware Type: <L40S>
  • [x] run make test to ensure correctness
  • [x] run make checkstyle to ensure code style
  • [ ] run make test-convergence to ensure convergence

vvvdwbvvv avatar Nov 25 '25 06:11 vvvdwbvvv