Yao Fehlis

INT4 quantization not working on MI210

1

INT8 quantization works fine, but INT4 does not work. 

Bandwidth achieved for INT8 is much smaller than FP16

3

I run CodeLlama 7B: when I use FP16, bandwidth achieved is 700 GB/s; however, when I use INT8, it is 197 GB/s. I run the model on one AMD MI210...

Code is extremely slow!

1

I am using AMD MI210s. After loading the models, the following steps are extremely slow (see screenshot). It turned out the Compilation time is 270 seconds. Could you please help...