worker-vllm

worker-vllm copied to clipboard

It seems there was a typo in the Dockerfile, preventing `$MODEL_REVISON` from ever being set since the build-arg was called `$MODEL_REVISON` but the ENV instruction tried to read a non-existing...

mikljohansson

How can i update to vLLM v0.4.1 for llama3 support ?

5

Hello everyone, I would like to update the vLLM version to v0.4.1 in order to get access to LLAMA3 but i don't know how modify the fork runpod/vllm-fork-for-sls-worker. Could you...

Lhemamou

vLLM 0.4.0, 30% base image size reduction

2

alpayariyak

Errors cause the instance to run indefinitely

19

Any errors caused by the payload cause the instance to hang in an error state indefinitely. You have to manually terminate the instance or you'll rack up a hefty bill...

gabewillen

I'm getting a `BadRequestError` when I try to test the vllm worker locally. I'm running my handler locally for testing, using `MODEL_NAME=/models/stablelm-3b-4e1t python3 -u /src/handler.py --rp_serve_api --rp_api_port 8000 --rp_api_host 0.0.0.0`,...

dpkirchner

I am looking to record input and output to the vLLM. I could put an HTTP proxy in front and capture the traffic, or modify your handler. Rather than make...

aodhan-domhnaill

Cannot load Tokenizers for some Models.

1

I am getting error saying it cannot load Tokenizers for some models like Yarn Mistral/Llama-2 models. Is there any reason why?

Mr-Nobody1

Thank you for this awesome repo 💯 . While building a custom revision, I noticed this typo. An incorrect model (main) gets downloaded in `download_model.py`

joennlae

OpenAI API: API errors have wrong HTTP code

4

Using a model that does not exist returns HTTP status 200, but the error message is in the JSON

lucasavila00

Any update on when this feature will be available? Thanks

joaomsimoes