Whether to support single card multi-instance loading

[dushulin]

one 3090-24g gpu, load multi instance, like triton

[hiworldwzj]

@dushulin You can try to use Docker to load multi instance. Lightllm can not load multi instance.

[dushulin]

thks, we deploy models on k8s

[XHPlus]

If you deploy LightLLM with K8S, you can directly utilize the native load balance ability of K8S infra, such as starting multiple pods in K8S.

[dushulin]

@XHPlus yes, you are right, but If you want to load multiple models on one gpu to share their computing power, the cost of using k8s vgpu technology may be too high. If you can directly deploy multiple models on a single card, it will be lighter