sglang icon indicating copy to clipboard operation
sglang copied to clipboard

add a version check for GGUF, according to the link in checkout

Open vincentzed opened this issue 8 months ago • 0 comments

Motivation

add the minimum version of gguf as needed.

Modifications

the version SET is based on the link to transformers that has a bit more context to the issue.

this is also related, which could also be worth checking out.

lastly we could extract in general utils, but this is optional, until we decide on the approach for VL2. this is also related, see https://github.com/sgl-project/sglang/blob/7c99103f4c2a5e92e34bbfc5f3779fe9a447c8ee/python/sglang/srt/configs/deepseekvl2.py#L82 lastly, one should also check out this for a bit more context: https://github.com/sgl-project/sglang/issues/4900

Specifically

We should sort out the dependencies in pyproject and classify them by vlm, quant, etc. Currently, because we don't pay attention to gguf in our daily test cases, we haven't tested whether gguf works with DeepSeek V3. @ mickqian @ yizhang2077

Checklist

  • [x] Format your code according to the Code Formatting with Pre-Commit.
  • [ ] Add unit tests as outlined in the Running Unit Tests.
  • [ ] Update documentation / docstrings / example tutorials as needed, according to Writing Documentation.
  • [ ] Provide throughput / latency benchmark results and accuracy evaluation results as needed, according to Benchmark and Profiling and Accuracy Results.
  • [ ] For reviewers: If you haven't made any contributions to this PR and are only assisting with merging the main branch, please remove yourself as a co-author when merging the PR.
  • [x] Please feel free to join our Slack channel at https://slack.sglang.ai to discuss your PR.

vincentzed avatar Apr 24 '25 00:04 vincentzed