AutoAWQ icon indicating copy to clipboard operation
AutoAWQ copied to clipboard

gemv_fast kernel only suppor 128 group_size

Open thincal opened this issue 10 months ago • 1 comments

It seems that only 128 group_size is supported by new gemv_fast kernel, is there any constrains for other group_size such as 64 ?

thincal avatar Apr 09 '24 02:04 thincal

I made one for you, so you can test for yourself (I was also curious).

GEMV - 64 groupsize (your requested parameters) solidrust/dolphin-2.8-mistral-7b-v02-AWQ-gemv-64gs

GEMM - 128 groupsize (standard AWQ quant) solidrust/dolphin-2.8-mistral-7b-v02-AWQ

suparious avatar Apr 11 '24 09:04 suparious