Liger-Kernel
Liger-Kernel copied to clipboard
linkedin
Efficient Triton Kernels for LLM Training
### 🚀 The feature, motivation and pitch This request is to adapt this to improve the training speed of Flux, a diffusion transformer. It's the top model on HuggingFace trending...
### 🚀 The feature, motivation and pitch Often used in pretraining of LMs for stabilization, i.e. the recent [Chameleon](https://arxiv.org/abs/2405.09818) & [PaLM](https://www.jmlr.org/papers/v24/22-1144.html). ### Alternatives [flash-attn](https://github.com/Dao-AILab/flash-attention/blob/main/flash_attn/ops/triton/cross_entropy.py) has implementations of abovementioned features, however,...
### 🐛 Describe the bug I've tried to use DoRA with LLaMA-Factory, but got this error `self and mat2 must have the same dtype, but got BFloat16 and Float` Haven't...
### 🚀 The feature, motivation and pitch We can leverage torch compile to fuse the things we cannot fuse now like upcasting, contiguous call, etc  Sample code: https://gist.github.com/Chillee/22cd93e11b887db1f596ab754d60a899#file-lce_benchmark-py Provided...
### 🐛 Describe the bug I'm using `flyte` to reproduce the token throughput and memory savings results reported in this [repo's README](https://github.com/linkedin/Liger-Kernel?tab=readme-ov-file#supercharge-your-model-with-liger-kernel) under slightly different conditions: using the `microsoft/Phi-3-mini-4k-instruct` model...
## Summary Add support for jamba model with Liger Kernel. The following ops can be patched with Liger kernel: - RMSNorm - cross_entropy - swiglu - lce_forward ## Testing Done...
## Summary Softcap support in FLCE to make it available for Gemma2 [#127](https://github.com/linkedin/Liger-Kernel/issues/127) ## Testing Done - Hardware Type: - [ ] run `make test` to ensure correctness - [...
### 🚀 The feature, motivation and pitch Create nice python autodoc which generating doc from python comments. Like https://triton-lang.org/main/index.html or https://github.com/vllm-project/vllm ### Alternatives _No response_ ### Additional context _No response_
## Summary ## Testing Done - Hardware Type: - [ ] run `make test` to ensure correctness - [ ] run `make checkstyle` to ensure code style - [ ]...
### 🚀 The feature, motivation and pitch Currently the most popular library might be https://github.com/databricks/megablocks. Would be interesting if we can implement it in triton and make it HF compatible...