hpcaitech
[FEATURE]: Is it Possible to integrate Liger-Kernel?
Describe the feature
https://github.com/linkedin/Liger-Kernel
Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.
Seems like a pretty light-weight library. cc @ver217 @isky-cd Any take on this? 😃
Liger Kernel is a collection of Triton kernels designed specifically for LLM training. It can effectively increase multi-GPU training throughput by 20% and reduce memory usage by 60%. We have implemented Hugging Face Compatible RMSNorm, RoPE, SwiGLU, CrossEntropy, FusedLinearCrossEntropy, and more to come.
I think this is a good attempt.
Does it compare with apex's implementation? We've integrate some apex cuda kernels and some of them are also implemented in Liger-kernel.
I think Apex only provides fused RMSNorm and LayerNorm kernels? They have some more
I think they are short-handed in wrapping up Zero Bubble, hybrid seq parallel and then they will focus on accelerate intergration? Feel free to ask other members to clarify further, but it'd be great if the community can make an initial PR on this, then we can help/comment. This is an open-source initiative after all, and we always welcome contributions🙂
Is there any documentation available on how to integrate a new kernel? So we can make a PR on this. Thanks a lot.
May I ask if you have a concept similar to Hugging Face's modeling_xxx for organizing modules? I have experience adding the liger-kernel there, and perhaps directly replacing the relevant modules could work.
