musoles

[Bug]: Impossible dependency requirement with GGUF

5

### Your current environment The output of `python env.py` Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch:...

[Bug]: Several errors when deploying GGUF models

5

### Your current environment The output of `python env.py` Collecting environment information... PyTorch version: 2.4.0+cu121 Is debug build: False CUDA used to build PyTorch: 12.1 ROCM used to build PyTorch:...

Documented example for autoscaling a multinode deployment

1

### 🚀 Feature Description and Motivation I can see in the documentation information for how to use autoscaling and how to do multi-node deployments, but not on both at the...

Autoscaling model deployments based on metrics

1

### ❓ Is your enhancement related to a problem? Currently it's possible to set a static number of replicas per model, but this is often leads to over-provisioning (underutilised replicas)...