docker NVILA inference error

[JIA-HONG-CHU]

I follow your instruction by running:

1. docker build -t vila-server:latest .
2. docker run --gpus all --ipc=host --ulimit memlock=-1 --ulimit stack=67108864
   -v ./hub:/root/.cache/huggingface/hub
   -it --rm -p 8000:8000
   -e VILA_MODEL_PATH=Efficient-Large-Model/NVILA-15B
   -e VILA_CONV_MODE=auto
   vila-server:latest
3. from openai import OpenAI

client = OpenAI( base_url="http://localhost:8000", api_key="MYKEY", ) response = client.chat.completions.create( messages=[ { "role": "user", "content": [ {"type": "text", "text": "What’s in this image?"}, { "type": "image_url", "image_url": { "url": "https://blog.logomyway.com/wp-content/uploads/2022/01/NVIDIA-logo.jpg", # Or you can pass in a base64 encoded image # "url": "data:image/png;base64,<base64_encoded_image>", }, }, ], } ], model="NVILA-15B", ) print(response.choices[0].message.content)

Encountered the error: openai.UnprocessableEntityError: Error code: 422 - {'detail': [{'loc': ['body', 'model'], 'msg': "unexpected value; permitted: 'VILA1.5-3B', 'VILA1.5-3B-AWQ', 'VILA1.5-3B-S2', 'VILA1.5-3B-S2-AWQ', 'Llama-3-VILA1.5-8B', 'Llama-3-VILA1.5-8B-AWQ', 'VILA1.5-13B', 'VILA1.5-13B-AWQ', 'VILA1.5-40B', 'VILA1.5-40B-AWQ'", 'type': 'value_error.const', 'ctx': {'given': 'NVILA-15B', 'permitted': ['VILA1.5-3B', 'VILA1.5-3B-AWQ', 'VILA1.5-3B-S2', 'VILA1.5-3B-S2-AWQ', 'Llama-3-VILA1.5-8B', 'Llama-3-VILA1.5-8B-AWQ', 'VILA1.5-13B', 'VILA1.5-13B-AWQ', 'VILA1.5-40B', 'VILA1.5-40B-AWQ']}}]}

[Lyken17]

oh, for NVILA the docker image and serving scripts is not updated to support yet. Please stay tuned and we will update an version in the coming week.

[wesen02]

Is there any news about the docker updates?

[Lyken17]

We have uploaded a new version of server.py in https://github.com/NVlabs/VILA/tree/main/serving to support NVILA models.

Docker build is on the way

[sriks6711]

Still waiting as of May. Should this issue be reopened?

Same error is repeating -

"... openai.UnprocessableEntityError: Error code: 422 - {'detail': [{'loc': ['body', 'model'], 'msg': "unexpected value; permitted: 'VILA1.5-3B', 'VILA1.5-3B-AWQ', 'VILA1.5-3B-S2', 'VILA1.5-3B-S2-AWQ', 'Llama-3-VILA1.5-8B', 'Llama-3-VILA1.5-8B-AWQ', 'VILA1.5-13B', 'VILA1.5-13B-AWQ', 'VILA1.5-40B', 'VILA1.5-40B-AWQ'", 'type': 'value_error.const', 'ctx' ..."