haystack-core-integrations
haystack-core-integrations copied to clipboard
CohereChatGenerator does not work for command r+ deployed on azure
Describe the bug
I deployed Cohere command r+ model on azure, but even if I set api_base_url
in CohereChatGenerator
it does not work. Interestingly, using OpenAIChatGenerator
does work but failed in tool-use related stuff.
To Reproduce
from haystack import Pipeline
from haystack.components.builders import DynamicChatPromptBuilder
from haystack.dataclasses import ChatMessage
from haystack_integrations.components.generators.cohere import CohereChatGenerator
from haystack.utils import Secret
import os
COHERE_API_KEY = os.environ.get("COHERE_API_KEY")
pipe = Pipeline()
pipe.add_component("prompt_builder", DynamicChatPromptBuilder())
pipe.add_component("llm", CohereChatGenerator(Secret.from_token(COHERE_API_KEY), api_base_url=os.environ.get('COHERE_BASE_URL')))
pipe.connect("prompt_builder", "llm")
location = "Berlin"
system_message = ChatMessage.from_system("You are an assistant giving out valuable information to language learners.")
messages = [system_message, ChatMessage.from_user("Tell me about {{location}}")]
res = pipe.run(data={"prompt_builder": {"template_variables": {"location": location}, "prompt_source": messages}})
print(res)
The error shows:
ApiError: status_code: 405, body: Method not allowed: POST https://127.0.0.1/chat
Describe your environment (please complete the following information):
- OS: Ubuntu 22.04.4 LTS
- Haystack version: 2.2.0
- Integration version: cohere-haystack 1.1.0