Yaron Rosenbaum
Yaron Rosenbaum
Same. Historically I've been using all sorts of databases successfully, but I haven't touched Docker for a couple of years, (so I'm quite rusty). I tried the above, same results....
Follow up, not sure if it's a separate bug: I changed the config yaml the model: from vllm/mistralai/Mistral-7B-Instruct-v0.2 to openai/mistralai/Mistral-7B-Instruct-v0.2. BTW the documentation is not clear on this. Now I...
litellm calls vllm (Mistral) with /chat/completions, which is not implemented: INFO: 10.42.20.72:43148 - "POST /chat/completions HTTP/1.1" 404 Not Found vLLM code shows they never implemented /chat/completions, just /v1/chat/completions and /v1/completions...
Following up: You suggested in a chat to add '/v1' to the base URL, and use 'openai' as a provider (even through I'm running vllm). This worked, except that prompts...
@psykhi Do you use the '/openai' prefix, or '/vllm' ? For me, using /vllm didn't work, so I had to switch to /openai, but then it doesn't translate prompts correctly...
@krrishdholakia Thank you, Can you please let me know how to handle prompt translation for models like Gemma? ( 'System prompt not supported')
Same issue here
Hey @krrishdholakia It was the latest docker image at the day I posted the comment
@krrishdholakia I can try, but I'm not sure exactly on which notebook this. I'm going through the langgraph academy online course, replacing all the OpenAI LLMs with Litellm. Or trying...