MoRA icon indicating copy to clipboard operation
MoRA copied to clipboard

Published 20 hours ago • verified publisher icon kongds

support QMoRA?

Open ByungKwanLee opened this issue 1 year ago • 1 comments

is it possible to support QMoRA with huggingface bitsandbytes?

ByungKwanLee avatar Jun 22 '24 18:06 ByungKwanLee

Thanks for your interest in our work.

MoRA has been implemented with Linear4bit in peft-mora is bnb.py, so it can run with a 4-bit model like LoRA. (But I haven't tried this before, so there may be some problems.)

kongds avatar Jun 24 '24 04:06 kongds