Sparsebit icon indicating copy to clipboard operation
Sparsebit copied to clipboard

Jst/support multi epoch pp qlora finetuning

Open Jiang-Stan opened this issue 1 year ago • 0 comments

Model 1epoch PPL 3epoch PPL
LLaMA-7b 2.397 2.345
LLaMA-65b 2.304

LLaMA-7b 4bit QLoRA(lr=3e-4) finetune loss曲线: 2023-05-12 14-27-44 的屏幕截图

LLaMA-65b 4bit QLoRA(lr=1e-4) finetune loss曲线(目前仅1 epoch): 2023-05-15 10-38-15 的屏幕截图

Jiang-Stan avatar May 08 '23 02:05 Jiang-Stan