sglang icon indicating copy to clipboard operation
sglang copied to clipboard

[WIP] Support NVLM-D

Open amosyou opened this issue 4 months ago • 0 comments

Motivation

NVLM-D is a decoder-only VLM released by NVIDIA recently that has great performance across different benchmarks. I've been trying to get a better understanding of VLM/LLM architectures, figured supporting a model would be a good start.

Modifications

NVLM-D is based off InternViT-6B as the vision encoder, and Qwen as the LLM. This should work out nicely since we can reuse the Qwen implementation. There's also some interesting modifications I'll need to make (I think) for their tile tags, and modify the image_processors.py file to accomodate a new VLM.

Checklist

  • [ ] Format your code according to the Contributor Guide.
  • [ ] Add unit tests as outlined in the Contributor Guide.
  • [ ] Update documentation as needed, including docstrings or example tutorials.

amosyou avatar Oct 07 '24 11:10 amosyou