Roger Wang

@sighingnow Sorry for the delayed response! I've merged main into your branch so the pre-commit error should be cleared. I'll enable ready status for this PR so at least we...

Thanks for this contribution! As we discussed offline, we'll be carefully reviewing this PR/design and think about how to enable end-to-end support for models like this with vLLM!

Can you remove the chat template and the `` token in your prompt and simply just do ``` python chat_completion_from_base64 = client.chat.completions.create( messages=[{ "role": "user", "content": [ { "type": "text",...

@arseniybelkov In fact, I don't think PaliGemma is supposed to work with OpenAI API format since it's never instruction fine-tuned and it never came with a chat-template. Can you try...

@arseniybelkov I don't think the model is supposed to work that way - could you try `caption es` as the prompt? (This is the same example given in the documentation...

> Yes, you're right, I didn't pay attention to it. I changed the prompt to `"caption en"` and now it gives me this > > ```shell > In this image...

@arseniybelkov Is it possible if you can send me the image and the prompt so I can repro and compare to `transformers` (especially on input embeddings)?

Hey @arseniybelkov I just want to let you know this is still on my list of TODO but I simply had limited bandwidth with other priorities. Sorry for the inconvenience!

Sorry for the delay - going to take a look at this PR tonight!

Looks like this PR doesn't work with https://github.com/huggingface/transformers/pull/36752 yet ``` ERROR 03-31 07:57:21 [core.py:377] File "/tmp-nvme/myenv/lib/python3.12/site-packages/transformers/processing_utils.py", line 1082, in from_pretrained ERROR 03-31 07:57:21 [core.py:377] return cls.from_args_and_dict(args, processor_dict, **kwargs) ERROR 03-31...