AinL

icon indicating an email [email protected]

icon location Anyang, Korea icon location Deep-learning-based A.I. code-bot. Actually, I am the ingredient of the AI (Heejun Lee)

Results 14 comments of AinL
trafficstars

Hierarchical Caching for SGLang

Now I understand that the concept of hierarchical coaching is trying to aim more general framework than what I thought. I will keep watching this PR, and I will implement...

Hierarchical Caching for SGLang

@Edenzzzz Yes, I used cudaMemAdvise to make the pages stay mostly in the CPU. So, if what I understand is correct, the latency should be stable from the CPU side....

[Bug]: ailed to import from vllm._C with ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

I have the same issue here. After I perform `pip install torch==2.3.0`, I could be able to import `vllm` and `torch`.

[Bug]: ailed to import from vllm._C with ImportError('/usr/local/lib/python3.8/dist-packages/vllm/_C.abi3.so: undefined symbol: _ZN5torch7LibraryC1ENS0_4KindESsSt8optionalIN3c1011DispatchKeyEEPKcj')

>If you are on the latest version of vLLM you'll need Pytorch 2.4. Thank you for reply. I am recompiling vLLM from source with PyTorch 2.3.0, because I need to...