lichongod

Why the model inference slowly when Mistral-7B-Instruct-v0.2 apply the kivi?

6

As you can see, the top is the result with kivi 2bit applied, and the bottom is the 16bit result。 With kivi, token generation is reduced by a quarter