Production deployment

[ArunAniyan]

Question

Hi, What is the best infrastructure and methodology to deploy llava for a production-grade application? Is a local application server like Ollama advisable? Do you know of other possible methods? Apart from ollama, llama.cpp is something that comes to mind. Have not tried triton-llm.

[nivibilla]

Sglang supports llava https://github.com/sgl-project/sglang