nano-vllm icon indicating copy to clipboard operation
nano-vllm copied to clipboard

readme: please state up-front: requires Linux for Triton and Nvidia GPU > 20 series

Open ai-bits opened this issue 2 months ago • 1 comments

Didn't expect that such a nano-demo would require an Nvidia GPU > 20 series and Linux (WSL) for Triton. Didn't read far /deep (pyproject.toml) enough, so I bumped into the Triton and then NV-GPU>20 series requirement.

Thanks for the "readable codebase"! G.

ai-bits avatar Sep 10 '25 19:09 ai-bits

The left-over, "ancient" 2070 was too old. (see above) I skipped over my dual RTX 4000 Ada and ran out for a 50-series for future experimentation with bfloat8, fp8, fp4,..

Now the Blackwell is too new to work with torch>=2.4.0 in your pyproject.toml. What speaks against bumping up to torch>=2.7.1 which comes with the required >=CUDA 12.6 for Blackwell?

Got things working now with brand-new torch 2.8.0. Thanks G.

ai-bits avatar Sep 14 '25 20:09 ai-bits