Bob

OOM when use InternVL2_5-1B-MPO

14

I followed the installation guide to build mldeploy (0.7.0.post3) from source. Inference using the PyTorch engine works fine. However, after quantizing the model to 4-bit using AWQ, I encountered an...