syncode icon indicating copy to clipboard operation
syncode copied to clipboard
verified publisher icon uiuc-focal-lab

Option to use flash_attention

Open Sindre01 opened this issue 8 months ago • 1 comments

It would be very usefull to have the option to use flash attention to increase speed and lower memory usage.

Sindre01 avatar Apr 03 '25 14:04 Sindre01

If you use SynCode as logits processor you should be able to use flash attention even in the current version. See this example

shubhamugare avatar Apr 08 '25 06:04 shubhamugare