Marut Pandya comments

Results 27 comments of


                                            Marut Pandya

Compatibility with deployment on a Pod?

Thanks for sharing the feedback. This worker currently supports the serverless. If you want to deploy on pods, it should be straight forward vllm deployment, let me know if I...

Unable to run gpt_oss model type

https://github.com/runpod-workers/worker-vllm/issues/210 @hoblin @Staberinde . Let me know, if you face any issues. I can take a look. Thanks.

Feature request: environment variable for `--task`

Sure. We can look into this.

Chat completion (template) not working with VLLM 0.6.3 + Serverless

I think setting CUSTOM_CHAT_TEMPLATE?

text generation sentences not in order (out of order)

Can you share your request payload?

text generation sentences not in order (out of order)

https://docs.runpod.io/serverless/workers/vllm/get-started, If you will scroll a bit down, you will find some sampling parameters to adjust, please try it with this.

Error: chat template not supported

@ParthKarth Did you try with custom_chat template ?