mlx-examples icon indicating copy to clipboard operation
mlx-examples copied to clipboard

interesting new finetuning approach from stanford - ReFT

Open fblissjr opened this issue 4 months ago • 0 comments

https://github.com/stanfordnlp/pyreft

uses flash attn and pyvene (https://github.com/stanfordnlp/pyvene) but don't see any specific kernels aside from flashattn. tried this on my cuda machine and it's neat - not sure how effective at scale yet, but worth exploring. anyone else looking into this?

fblissjr avatar Apr 06 '24 15:04 fblissjr