gpt-4o equivalent model doesn't answer properly to text inputs (using docker aio-cpu images)

[M0Rf30]

LocalAI version: 2.25.0-aio-cpu container image and all the versions that adopt bakllava-1 as gpt-4o alias

Environment, CPU architecture, OS, and Version: Arch Linux and Fedora CoreOS

Describe the bug aio-cpu sets bakllava-1 as vision model (gpt-4o). I tried it in many different releases and it always answers in weird ways with no logical correlation with the user input.

To Reproduce Just select the gpt-4o model in Chat section (webui) and write "Hi" or something else. Images evaluation seems to not suffer of this. If I upload an image and ask for details about the image, it will answer properly.

Expected behavior I expect reasonable answers for both normal text input and images.

Suggestions Look for other omni models to use for vision. Or maybe is a prompt issue?

[ejalal]

Same here, installed localai:latest-cpu image and manually dowloaded qwen2.5-coder-14b model from the webui

docker run -p 8080:8080 --name local-ai -ti -v $PWD/models:/build/models localai/localai:latest-cpu

[nick-pape]

@M0Rf30 try adding some stopwords, the config for GPT-4o (llava-v1.6-mistral-7b.Q5_K_M) doesn't come with any by default (at least on CUDA). I added these two blocks from the gpt-4 config and things started working well. (Not sure that the functioncall was needed though). You may need to adapt for the bakllava models.

https://github.com/mudler/LocalAI/issues/4816

[github-actions[bot]]

This issue is stale because it has been open 90 days with no activity. Remove stale label or comment or this will be closed in 5 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 5 days with no activity.