llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

Published 20 hours ago • verified publisher icon ggml-org

Add avx-512 support?

Open FNsi opened this issue 1 year ago • 6 comments

No clue but I think it may work faster

FNsi avatar Mar 15 '23 12:03 FNsi

No clue but I think it may work faster

On i7-11700F, when I built with AVX512 flags in VS2022, my speed increased from ~0.37 s/token to ~0.34 s/token. Don't know if it was placebo or not. Maybe a proper assembly inline code will be better

congdm avatar Mar 15 '23 20:03 congdm

No clue but I think it may work faster

On i7-11700F, when I built with AVX512 flags in VS2022, my speed increased from ~0.37 token/s to ~0.34 token/s. Don't know if it was placebo or not. Maybe a proper assembly inline code will be better

How a single flag can make 10% boost? Seems great!

FNsi avatar Mar 16 '23 00:03 FNsi

How a single flag can make 10% boost? Seems great!

It's actually placebo :) Now I'm testing again, they're both running at the same speed

congdm avatar Mar 16 '23 01:03 congdm

How a single flag can make 10% boost? Seems great!

It's actually placebo :) Now I'm testing again, they're both running at the same speed

😂hahah, that make more sense

FNsi avatar Mar 16 '23 01:03 FNsi

This would require code changes, not just compiler flags.

RazeLighter777 avatar Mar 17 '23 11:03 RazeLighter777

Interesting https://stackoverflow.com/questions/63484266/enabling-avx512-support-on-compilation-significantly-decreases-performance

dzid26 avatar Mar 22 '23 04:03 dzid26