[BUG] Conversational Agent Throws Exception When Used With Chain Tool, Retrieval QA Chain & Open Source Model

[DebajitKumarPhukan]

Describe the bug Conversational agent when used along with a chain tool backed by Retrieval QA chain and an open source LLM like PALM 2 doesn't work properly and 100% of time fail to fetch data from vector store.

To Reproduce

1. Do a set up as shown below in Flowise UI.
2. Put a csv file having a customer id, name and phone number.
3. Ask the same question as shown in the screenshot below.

Expected behavior The conversational agent should be able to use the customer-qna chain tool and fetch the results by querying from vector store.

Screenshots

Flow Added the chat flow json file in order to help you reproduce the issue. Google Palm Flow - Chatflow.json

Setup

• Installation [yarn start]
• Flowise Version [e.g. 1.3.8]
• OS: [Windows]
• Browser [chrome]

Additional context

1. The result is same even if I replace "Retrieval QA Chain" with "Conversational Retrieval QA Chain".
2. I haven't tried out with Open AI as I don't have any investment done for it.

Resolving this issue will unlock more areas for open source models via Flowise.

[HenryHengZJ]

I suggest create a flow first to upsert the docs to vector store. Then use Conversational Retrieval Agent template to retrieve from vector store:

[DebajitKumarPhukan]

@HenryHengZJ First of all, thanks for looking into it. The primary objective is to use "tools" in the flow and not just a simple QnA flow. The main issue is that the "open source model" (like PALM or Hugging Face model) don't work well with a combo of an agent like "Conversational agent", a retrieval chain like "Retrieval QA Chain" or "Conversational Retrieval QA Chain" and open source models.

I believe that if the open source model is replaced with Open AI model (for which I don't have any paid key), this might work, but this completely defeats the purpose of Flowise to be model agnostic.

[HenryHengZJ]

yeah i agree. Most open source models dont work with agent/complex chain because the underlying prompt is tailored more to ChatGPT. In that case, try using Prompt Chaining with VectorStore template, which means create your own PromptTemplate and LLMChain and use that as a ChainTool

[DebajitKumarPhukan]

@HenryHengZJ Is there any agent component at present I can use from Flowise which works well with open source model ?. If you have any chatflow example for it, that would be really great !