markouustalu

### 🐛 Describe the bug Ran on 1x or 2x 3060 12GB, prompt was single one sentence coding instruction for a sample program While there is speed reduction with vLLM...

[Feature]: Automatic max-model-len or max-num-seqs

3

### 🚀 The feature, motivation and pitch **Feature Request:** Automatically Adjust --max-model-len and --max-num-seqs Based on GPU Memory, Cache Size, and Other Parameters **Problem to Solve:** Currently, maximizing GPU memory...