Nipun Garg comments

Repositories
Issues
Comments

Results 3 comments of


                                            Nipun Garg

Attribute Error: /llama.cpp/libllama.so: undefined symbol: llama_numa_init

Getting this error on aarch64 as well Traceback (most recent call last): File "", line 198, in _run_module_as_main File "", line 88, in _run_code File "/usr/local/lib/python3.11/site-packages/uvicorn/__main__.py", line 4, in uvicorn.main()...

[Performance] use redis and two vllm instance, second request is slower than first

My understanding is if we use Redis as KV cache offloading and sharing, then different instances of vLLM(be it on the same node or different), then these 2 instances should...

Add ability to disable FIPS in the container, when running on FIPS enabled host

Upvote on the ability to run a non-FIPS container on a FIPS-enabled host.