Santhana Mikhail Antony S comments

Results 16 comments of


                                            Santhana Mikhail Antony S

[Bug]: Unable to infer QLoRA adapter using vLLM Docker

> see: [#4186 (comment)](https://github.com/vllm-project/vllm/issues/4186#issuecomment-2241307064) But if I merge the LoRA adapter, I cannot use multiple LoRAs, right?

DOC: Remove outdated versionadded/changed directives?

Hey Antony here. I am new to numpy. Will change as many files as possible and raise a PR.

[BUG] Summarize fails even when a model response is generated with the error "HTTP request failed: POST predict: Post "http://127.0.0.1:36333/completion": EOF"

any fix for this issue?

Server error: transport error

Getting this same error when I try to hit a loaded model from the start with A40 :(

Latest Docker image fails while initializing gemma2

> Ah it seem like this one doesn't have softcapping (#2273). I'd recommend using the latest TGI version. > > Would that work for you? Can you take a look...

ChatHuggingFace + HuggingFaceEndpoint does not properly implement `max_new_tokens`

> I think this is a `ChatHuggingFace` bug. > > You have to call `bind` on the chat object with the parameters you want changed before running `invoke`. They do...