Song

[BUG] Fail to inference with 8bit quantized bloom-3b model, shape mismatch error

10

**Describe the bug** Hi, I am trying to run a bloom3b model with 8bit quantization with two 2080Ti GPUs. Because there exists no pre-quantized version, I simply change the `dtype`...