AutoAWQ
AutoAWQ copied to clipboard
gemv_fast kernel only suppor 128 group_size
It seems that only 128 group_size is supported by new gemv_fast kernel, is there any constrains for other group_size such as 64 ?
I made one for you, so you can test for yourself (I was also curious).
GEMV - 64 groupsize (your requested parameters) solidrust/dolphin-2.8-mistral-7b-v02-AWQ-gemv-64gs
GEMM - 128 groupsize (standard AWQ quant) solidrust/dolphin-2.8-mistral-7b-v02-AWQ