llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard
Published 2 months ago • verified publisher icon ggml-org

AMX isa Native addition

Open abhilash1910 opened this issue 2 years ago • 3 comments

From thread https://github.com/ggerganov/llama.cpp/issues/2555 Initial support for AMX bf16 isa for build. @ggerganov could you take a look ? Thanks

abhilash1910 avatar Oct 25 '23 12:10 abhilash1910

Question: Is the AMX enabling only need to add compiler option? Without change any gmm code for tile operation? Thanks! Have you test whether AMX_BUSY found via perf when running ?

kenplusplus avatar Feb 17 '24 14:02 kenplusplus

@abhilash1910 Do you plan in implementing AMX support? AFAICT this just adds build flags, but those would be useless without actual implementation

ggerganov avatar Feb 18 '24 16:02 ggerganov

@ggerganov yes this is a future plan , and I will continue probably post we complete the development plans in terms of multi gpu support on sycl . I will mark it as draft for now.

abhilash1910 avatar Feb 19 '24 02:02 abhilash1910

I'm really looking forward to AMX support. It will significantly increase the performance.

nai-kon avatar Mar 19 '24 07:03 nai-kon