lion-pytorch
lion-pytorch copied to clipboard
lucidrains
add an 8-bit version with bitsandbytes
https://github.com/TimDettmers/bitsandbytes/blob/main/compile_from_source.md
recent results have tipped me over to recommending this optimizer under the condition of high batch sizes (and mainstream architectures). will put some more work into it
if there are any CUDA / 8-bit optimizer experts in the crowd, would welcome a code review over at the bitsandbytes repository
@lucidrains Is there a reason behind the 64 per_device_train_batch_size threshold recommendation? With consumer 24gb gpus, it is almost impossible to train or finetune any meaningful model with that high batch size to take advantage of lion. It would be nice if lion is applicable to smaller batches? Thanks
The original authors note that Lion likely doesn't show significant gains over AdamW under the small batch (<64) regime. See https://arxiv.org/pdf/2302.06675.pdf
