bug: OpenAI Model doesnt work with Retrieval

[jefffortune]

Describe the bug Good morning, I am having issues when using Mistral API, I am overriding it with my own mistral API and when chatting it works great and as expected. When I try retrieval it will just hang and become unresponsive when it trying index the document embeddings.

If I switch back to a Jan downloaded model it works just fine.

Expected behavior I expected that Jan would embed the document.

Environment details

• Operating System: MacOs
• Jan Version: 0.4.11
• Processor:Apple M3
• RAM: 32

Logs If the cause of the error is not clear, kindly provide your usage logs:

• tail -n 50 ~/jan/logs/app.log if you are using the UI
• tail -n 50 ~/jan/logs/server.log if you are using the local api server 2024-04-17T13:43:31.082Z [SPECS]::Machine: arm64 2024-04-17T13:43:31.082Z [SPECS]::OS Platform: darwin 2024-04-17T13:43:31.082Z [SPECS]::CPUs: [{"model":"Apple M3 Pro","speed":24,"times":{"user":12631320,"nice":0,"sys":11478910,"idle":101497290,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":10574120,"nice":0,"sys":8726620,"idle":106907310,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":6905210,"nice":0,"sys":5239540,"idle":114866090,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4563210,"nice":0,"sys":3041250,"idle":119972560,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":2953650,"nice":0,"sys":1838530,"idle":123093440,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":2073730,"nice":0,"sys":1238160,"idle":124726500,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4732610,"nice":0,"sys":1312450,"idle":122046270,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4777050,"nice":0,"sys":1338410,"idle":121985740,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4643840,"nice":0,"sys":1300060,"idle":122170460,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4694710,"nice":0,"sys":1318950,"idle":122102450,"irq":0}},{"model":"Apple M3 Pro","speed":24,"times":{"user":4728540,"nice":0,"sys":1329190,"idle":122061190,"irq":0}}] 2024-04-17T13:43:31.082Z [SPECS]::OS Release: 23.4.0 2024-04-17T13:43:31.082Z [NITRO]::Debug: Request to kill Nitro 2024-04-17T13:43:31.082Z [SPECS]::Parallelism: 11 2024-04-17T13:43:31.082Z [SPECS]::Free Mem: 509493248 2024-04-17T13:43:31.082Z [SPECS]::Total Mem: 38654705664 2024-04-17T13:43:31.116Z [NITRO]::Debug: Nitro process is terminated

Additional context There noting in logs of importance to showing the error. I am guessing it trying to have the LLM make embeddings but it cannot since all it able do is serve responses.

[haliliceylan]

If I switch back to a Jan downloaded model it works just fine.

Which model are you using for retrieval successfully? Can you give more detailed information?

[jefffortune]

@haliliceylan if I use Mistral Instruct 7B Q4 from Jan.ai model hub it works as expected. If I provide the OpenAI settings with my URL the hosted env in AWS and API_key it will not do retrieval operations just chat.

[hahuyhoang411]

Hi @jefffortune and @haliliceylan , Thank you for using Jan. Currently, RAG is still in experimental feature. We will update it to work properly with Remote API in the next few sprints. Please stay tuned.

For now, the best way to use RAG in Jan is to use the local host model (e.g. Llama-3 8B, Mistral 7B)

[Van-QA]

Although it's experimental, dev team did a quick fix recently which hopefully will resolve the issue of RAG with OpenAI. If possible, you can check it out on Jan nightly v0.4.11-392 🙏 Many thanks

[jefffortune]

@Van-QA I downloaded the nightly fix and try it out.

My steps:

• Jan nightly v0.4.11-392
• I am using VLLM as the openai compatible server
• I used a mistral instruct finetune model
• Copy the OpenAI model folder from jan and updated values to have it populate in UI. Restarted Jan.
• Updated values in OpenAI config for URL and API key
• Started new chat and ensured my finetune was selected and retrieval was active.
• Uploaded a small pdf and asked it to summarize.
• Jan.AI could not find document but it didnt freeze this time like before.