Venkat Raman

Hello @rstojnic, Hope you are doing well. Looking forward to your PR review.

@navneet1v Nice ! I was looking for this and came across your PR. I would like to propose making k-NN search as an optional feature that could be controlled via...

@navneet1v Thanks for your quick response. Your approach sounds good 👍🏽

@vamshin @navneet1v I'm not from OpenAI / maintainer of this repo. I'm an OpenSearch community user as well and was trying out your change while pending review. (Approved to not...

Hey @martinigoyanes , Just taking a stab at the issue here. Without knowing actual model config, as per [this](https://www.anyscale.com/blog/continuous-batching-llm-inference) article: `The amount of GPU memory consumed scales with the base...

Good question. In transformer inference, both `prefill` & `decode` phase use GPU VRAM. i.e., processing is done by moving model weights, kvcache, embeddings, etc., from VRAM to L2, L1 &...

No problem @martinigoyanes. vLLM supports this: https://github.com/vllm-project/vllm/issues/2304 Maybe TGI already supports this natively or through vLLM integration ? I have to look into TGI config to have more clarity on...

> I think TGI supports TP when model does not fit on 1 GPU but it does not allow you to force it to happen even when model fits in...

Hey @martinigoyanes , > you can leverage 100% VRAM from the extra GPU Maybe there is a terminology/communication gap here. The above statement is not correct. Higher throughput is achieved...

> When serving LLMs for "real" use cases, you must put some kind of rate limiter in front of it `vLLM`, `TGI` & `Triton` are already powering several "real" use...