sglang
sglang copied to clipboard
[WIP] Support NVLM-D
Motivation
NVLM-D is a decoder-only VLM released by NVIDIA recently that has great performance across different benchmarks. I've been trying to get a better understanding of VLM/LLM architectures, figured supporting a model would be a good start.
Modifications
NVLM-D is based off InternViT-6B as the vision encoder, and Qwen as the LLM. This should work out nicely since we can reuse the Qwen implementation. There's also some interesting modifications I'll need to make (I think) for their tile tags, and modify the image_processors.py
file to accomodate a new VLM.
Checklist
- [ ] Format your code according to the Contributor Guide.
- [ ] Add unit tests as outlined in the Contributor Guide.
- [ ] Update documentation as needed, including docstrings or example tutorials.