eleyzhu

"torch.cuda.OutOfMemoryError: CUDA out of memory" when deploy LLM with TGI in Kubernetes cluster

1

When deploy the LLM with TGI in k8s cluster with the pod like below: apiVersion: v1 kind: Pod metadata: name: text-generation-inference labels: run: text-generation-inference spec: containers: - name: text-generation-inference image:...