Nicolas Casademont

Results 5 comments of Nicolas Casademont

[Qwen/Qwen2.5-14B-Instruct-GPTQ-Int8] Bad Responses with High Concurrent Requests

I'm having the same issue with a r1-distilled-qwen-7b on h100s. Weirdly running it with version 3.0.1 works perfectly fine.

[Bug]: Docker vLLM 0.9.1 CUDA error: an illegal memory access, sampled_token_ids.tolist()

Hello, just in case this might help, I'm having a similar issue with mistralai/Mistral-Small-3.1-24B-Instruct-2503, vllm 0.9.1 and 1 H100 80GB. I'm testing the fix mentioned above and will report back.

Tool Calling using Vercel's AI SDK not working as intended

👋 I made a PR (#2954) for a potential fix. I'm hoping it fixes the issues you were having, if any of you want to test it with your use...

Structured output doesn't work with open ai endpoint

Hello, I have a PR for just that fix if you want to take a look: #2954.

Error With Tool Calling

👋 I made a PR (#2954) for a potential fix. I'm hoping it fixes the issues you were having, if any of you want to test it with your use...