support fastchat openai api?

[seanzhang-zhichen]

support fastchat openai api?

how do?

[anakin87]

Hey, @seanzhang-zhichen!

Please add some details to this issue:

• What would you like to do? What is your use case?
• Can you add a code snippet to explain your reqyest?
• Can you link some relevant references?

[seanzhang-zhichen]

Hey, @seanzhang-zhichen!

Please add some details to this issue:

• What would you like to do? What is your use case?
• Can you add a code snippet to explain your reqyest?
• Can you link some relevant references?

I deployed a qwen-72b model using fastchat's openai api server, but I don't know how to use it in haystack, I need to customize base_url, api_key and headers.

[seanzhang-zhichen]

mycode:

from haystack.nodes import PromptNode, PromptTemplate, AnswerParser, BM25Retriever, PromptModel
from haystack.pipelines import Pipeline
from haystack.utils import print_answers
from haystack.agents import Tool
from haystack.nodes.prompt.invocation_layer.open_ai import OpenAIInvocationLayer

# model = "Qwen-72B-Chat-Int4"
model_name = "Qwen-72B-Chat-Int4"
api_key = "sk-EMPTY"
base_url = "http:/xxxx/v1"
headers = {
    'Content-Type': 'application/json',
    "Authorization": "Basic xxx"
}

OpenAIInvocationLayer.headers = headers
OpenAIInvocationLayer.url = "http://xxx/v1/completions"


generative_pipeline = Pipeline()


search_tool = Tool(
    name="seven_wonders_search",
    pipeline_or_node=generative_pipeline,
    description="useful for when you need to answer questions about the seven wonders of the world",
    output_variable="answers",
)


from haystack.nodes import PromptNode

model = PromptModel(model_name_or_path=model_name, 
    invocation_layer_class=OpenAIInvocationLayer, 
    api_key=api_key,
    model_kwargs={
        "headers": headers,
        "api_base": base_url
    }
)

agent_prompt_node = PromptNode(
    model_name_or_path=model,
    api_key=api_key,
    stop_words=["Observation:"],
    debug=True,
)



agent_prompt = """
In the following conversation, a human user interacts with an AI Agent. The human user poses questions, and the AI Agent goes through several steps to provide well-informed answers.
The AI Agent must use the available tools to find the up-to-date information. The final answer to the question should be truthfully based solely on the output of the tools. The AI Agent should ignore its knowledge when answering the questions.
The AI Agent has access to these tools:
{tool_names_with_descriptions}

The following is the previous conversation between a human and The AI Agent:
{memory}

AI Agent responses must start with one of the following:

Thought: [the AI Agent's reasoning process]
Tool: [tool names] (on a new line) Tool Input: [input as a question for the selected tool WITHOUT quotation marks and on a new line] (These must always be provided together and on separate lines.)
Observation: [tool's result]
Final Answer: [final answer to the human user's question]
When selecting a tool, the AI Agent must provide both the "Tool:" and "Tool Input:" pair in the same response, but on separate lines.

The AI Agent should not ask the human user for additional information, clarification, or context.
If the AI Agent cannot find a specific answer after exhausting available tools and approaches, it answers with Final Answer: inconclusive

Question: {query}
Thought:
{transcript}
"""


from haystack.agents import AgentStep, Agent


def resolver_function(query, agent, agent_step):
    return {
        "query": query,
        "tool_names_with_descriptions": agent.tm.get_tool_names_with_descriptions(),
        "transcript": agent_step.transcript,
        "memory": agent.memory.load(),
    }


from haystack.agents.base import Agent, ToolsManager

conversational_agent = Agent(
    agent_prompt_node,
    prompt_template=agent_prompt,
    prompt_parameters_resolver=resolver_function,
    tools_manager=ToolsManager([search_tool]),
)

conversational_agent.run("你好")

[anakin87]

I imagine your code is not working... What is the error you get?

[seanzhang-zhichen]

Agent custom-at-query-time started with {'query': 'tell me seven wonders of the world', 'params': None} Maximum number of iterations (8) reached for query (tell me seven wonders of the world). Increase max_steps or no answer can be provided for this query.

[anakin87]

The error does not seem related to a problem in contacting the endpoint. You can try increasing the max number of iterations (max_steps) and see what happens.

[seanzhang-zhichen]

The error does not seem related to a problem in contacting the endpoint. You can try increasing the max number of iterations (max_steps) and see what happens.

Can you provide a sample using a customized fastchat openai api