nano-vllm
nano-vllm copied to clipboard
readme: please state up-front: requires Linux for Triton and Nvidia GPU > 20 series
Didn't expect that such a nano-demo would require an Nvidia GPU > 20 series and Linux (WSL) for Triton. Didn't read far /deep (pyproject.toml) enough, so I bumped into the Triton and then NV-GPU>20 series requirement.
Thanks for the "readable codebase"! G.
The left-over, "ancient" 2070 was too old. (see above) I skipped over my dual RTX 4000 Ada and ran out for a 50-series for future experimentation with bfloat8, fp8, fp4,..
Now the Blackwell is too new to work with torch>=2.4.0 in your pyproject.toml. What speaks against bumping up to torch>=2.7.1 which comes with the required >=CUDA 12.6 for Blackwell?
Got things working now with brand-new torch 2.8.0. Thanks G.