Add support for LISA ?

[risedangel]

Hello any plans to support with LISA ?

Arxiv: https://arxiv.org/pdf/2403.17919.pdf

how it compares in the terms of VRAM usage ? standard 16 bit model fine tuning ?

[risedangel]

https://github.com/OptimalScale/LMFlow this tool seems to suppot it.

[danielhanchen]

Oh yep had a discussion with some researchers about this! Speed wise, because the first and the last get updated, the gradients have to be backpropagated to the start, so not that much faster than LoRA.

A big issue is the benchmarks which I'll have to manually check - I'm slightly skeptical of LoRA being "worse" than FT / LISA or LISA being better than full finetuning - very counterintuitive and confusing actually.

[risedangel]

@danielhanchen hi, any news?

[danielhanchen]

@risedangel No sorry :( Been stuck on fixing bugs