Liger-Kernel
Liger-Kernel copied to clipboard
linkedin
Efficient Triton Kernels for LLM Training
## Summary ## Testing Done - Hardware Type: - [ ] run `make test` to ensure correctness - [ ] run `make checkstyle` to ensure code style - [ ]...
### 🚀 The feature, motivation and pitch - [ ] 1. Current GRPO assume KL term is added and advantage is computed inside the loss, we wanna open this to...
### 🚀 The feature, motivation and pitch This model from Upstage is extremely strong for models that fit on a single GPU for training and inference! https://huggingface.co/upstage/solar-pro-preview-instruct. However, it does...
### 🐛 Describe the bug # State of Liger Kernel CI on MI300X on ROCm 6.3 ## Environment: * `python3 -m pip list | grep triton` ``` pytorch-triton-rocm 3.3.0+git96316ce5 triton...
### 🚀 The feature, motivation and pitch ByteDance [DAPO](https://dapo-sia.github.io/) is the open-sourced SOTA RL algorithm that achieves 50 points on AIME 2024 based on the Qwen2.5-32B pre-trained model, surpassing the...
Hello Liger Kernel team, First of all, thank you for making this project available! I’ve been exploring your codebase and tried to implement GKDTrainer using the chunked_jsd_loss similarly to how...
### 🐛 Describe the bug It is normal when using fp16, but there are some problems with bf16. It seems that because swiglu uses tl.sigmoid(), it needs to be converted...
### 🚀 The feature, motivation and pitch Thank you for developing such an outstanding library. With its help, my pre-training work has achieved an MFU of 51%. Truly appreciate your...
### 🚀 The feature, motivation and pitch We want to support various alignment and distillation loss functions. Refer this PR on ORPO: #362 ## Progress ### Alignment - [x] ORPO...
### 🐛 Describe the bug Getting an error when I try to build from source on ROCM ### Reproduce ``` pip install torch torchvision torchaudio --index-url https://download.pytorch.org/whl/rocm6.2 git clone https://github.com/linkedin/Liger-Kernel.git...
