litgpt
litgpt copied to clipboard
WIP/Add: NEFTune to Lit-GPT
Picking up an issue I opened last week, feel free to close if no interest or being done elsewhere.
So far I've added it to the LoRA implementation, but could use it as well on the full fine-tuning too.
Lmk!
https://github.com/Lightning-AI/lit-gpt/issues/642