mku-wedoai
Results
2
comments of
mku-wedoai
I would love to see EXL2 support in vLLM!
I experienced the issue too. My setup: - 2 x A100 80 GB, run in Kubernetes cluster, - vLLM version: 0.5.4, - model: Mixtral-8x22B-Instruct-v0.1-GPTQ-4bit, - start arguments: ``` - "--tensor-parallel-size"...