MultiRetrievalQAChain requires ChatModel... but should it?

[k3nnethfrancis]

System Info

python 3.10. Langchain 0.0.184

Seems that MultiRetrievalQAChain is requiring a chat model to run, but feels like this shouldn't be required. Other QA pipelines don't require a chat model, and I don't see why they should. I'm guessing this is just a symptom of some parts of the system being upgraded to use chatmodels, since OpenAI is used for a lot of little things under the hood with langchain, like deciding which vectorDB to use based on the retriever_infos dict. However, if you are trying to use another LLM aside from openAI, you cannot use MultiRetrievalQA.

Further, I've run into some issues with this sort of thing across otherQA pipelines when using huggingface LLMs. Generally, better support for huggingface is important to me, and I think would be important to many others in the near future as the open source model ecosystem continues to grow.

Who can help?

No response

Information

• [ ] The official example notebooks/scripts
• [ ] My own modified scripts

Related Components

• [X] LLMs/Chat Models
• [X] Embedding Models
• [ ] Prompts / Prompt Templates / Prompt Selectors
• [ ] Output Parsers
• [ ] Document Loaders
• [X] Vector Stores / Retrievers
• [ ] Memory
• [ ] Agents / Agent Executors
• [ ] Tools / Toolkits
• [ ] Chains
• [ ] Callbacks/Tracing
• [ ] Async

Reproduction

import langchain from langchain.llms import HuggingFacePipeline, HuggingFaceHub from langchain.embeddings import HuggingFaceEmbeddings from langchain.schema import Document from langchain.document_loaders import DirectoryLoader, TextLoader from langchain.indexes import VectorstoreIndexCreator from langchain.vectorstores import Chroma from langchain.text_splitter import CharacterTextSplitter from langchain.chains import RetrievalQA from langchain.chains.router import MultiRetrievalQAChain from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM, GPTJForCausalLM, AutoModelForQuestionAnswering, GPTJForQuestionAnswering from abc import ABC, abstractmethod from typing import List, Dict, Optional import os import requests import shutil

#device object device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu')

hf embeddings model

embeddings = HuggingFaceEmbeddings(model_name="sentence-transformers/all-mpnet-base-v2")

Cerebras GPT2.7B

model = AutoModelForCausalLM.from_pretrained( "cerebras/Cerebras-GPT-2.7B", torch_dtype=torch.float16, ).to(device)

tokenizer = AutoTokenizer.from_pretrained("cerebras/Cerebras-GPT-2.7B")

hf pipeline

gen_pipe = pipeline( "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=200, early_stopping=True, no_repeat_ngram_size=2, device=0 )

define llm

gen_llm = HuggingFacePipeline(pipeline=gen_pipe)

load documents and create retrievers

docs1 = TextLoader('docs/dir1/mydoc1.txt').load_and_split() retreiver1 = Chroma.from_documents(docs1, embeddings).as_retriever()

docs2 = TextLoader('docs/dir2/mydoc2.txt').load_and_split() retriever2 = Chroma.from_documents(docs2, embeddings).as_retriever()

retriever_infos = [ { "name": "space-knowledge", "description": "Good for answering general questions about outer space.", "retriever": retriever1 }, { "name": "earth-knowledge", "description": "Good for answer questions about planet earth", "retriever": retriever2 } ]

chain = MultiRetrievalQAChain.from_retrievers(llm=gen_llm, retriever_infos=retriever_infos, verbose=True)

Expected behavior

I expect this to work the same way it does when using an openAI model. i.e., the model chooses the vectorDB to use based on the question, searches, and returns a response.

[johnbrisbin]

MultiRetrievalQAChain imports ChatOpenAI for no apparent reason in this line (16): from langchain.chat_models import ChatOpenAI This errant import creates a chain of errors at runtime if no OpenAI modules are otherwise used and a non-OpenAI llm is specified.

1. Failed to set OPENAI_API_KEY (validation)
2. Failed to import OpenAI

[shubham184]

Also issue while using AzureOpenAI

[iljoong]

I'm using Azure OpenAI and I had the same issue.

Here is my solution. You can specify default_chain in MultiRetrievalQAChain.from_retrievers. see following code snippet.

llm = AzureChatOpenAI(temperature=0.0, deployment_name='gpt-35-turbo')

# https://github.com/hwchase17/langchain/blob/master/langchain/chains/conversation/prompt.py
DEFAULT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
{history}
Human: {input}
AI:"""

prompt_default_template = DEFAULT_TEMPLATE.replace('input', 'query')

prompt_default = PromptTemplate(
    template=prompt_default_template, input_variables=['history', 'query']
)
default_chain=ConversationChain(llm=llm, prompt=prompt_default, input_key='query', output_key='result')

...

chain = MultiRetrievalQAChain.from_retrievers(llm, retriever_infos, default_chain=default_chain, verbose=True)

[shubham184]

I'm using Azure OpenAI and I had the same issue.

Here is my solution. You can specify default_chain in MultiRetrievalQAChain.from_retrievers. see following code snippet.

llm = AzureChatOpenAI(temperature=0.0, deployment_name='gpt-35-turbo')

# https://github.com/hwchase17/langchain/blob/master/langchain/chains/conversation/prompt.py
DEFAULT_TEMPLATE = """The following is a friendly conversation between a human and an AI. The AI is talkative and provides lots of specific details from its context. If the AI does not know the answer to a question, it truthfully says it does not know.

Current conversation:
{history}
Human: {input}
AI:"""

prompt_default_template = DEFAULT_TEMPLATE.replace('input', 'query')

prompt_default = PromptTemplate(
    template=prompt_default_template, input_variables=['history', 'query']
)
default_chain=ConversationChain(llm=llm, prompt=prompt_default, input_key='query', output_key='result')

...

chain = MultiRetrievalQAChain.from_retrievers(llm, retriever_infos, default_chain=default_chain, verbose=True)

@iljoong when i do chain.run getting OutputParserException Traceback (most recent call last) File ~\.conda\envs\mydemoenv\Lib\site-packages\langchain\chains\router\llm_router.py:80, in RouterOutputParser.parse(self, text) 79 expected_keys = ["destination", "next_inputs"] ---> 80 parsed = parse_json_markdown(text, expected_keys) 81 if not isinstance(parsed["destination"], str):

[iljoong]

@shubham184, It looks like it's a different error, that is, parsing error. Your questions or description in retriever_infos may produce an unexpected output.

Try this example first: https://github.com/hwchase17/langchain/blob/master/docs/modules/chains/examples/multi_retrieval_qa_router.ipynb

To test, use following question first. This will give an error for Azure OpenAI as an llm.

print(chain.run("tell me about Issac Newton"))

Add above code snippet to the example and ask the same question. This will work.

I also recommend to use debug mode and exam how langchain produces intermediate steps.

import langchain
langchain.debug = True.

[shubham184]

issue continues, so MultiRetrievalQAChain works absolutely fine when using TextLoader pg_docs = TextLoader('AutoVector\\docs\\pge.txt').load_and_split()

`> Entering new MultiRetrievalQAChain chain...
pg essay: {'query': 'What does Paul Graham regret about his work?'}
> Finished chain.
"The context mentions that Paul Graham regrets creating Hacker News (HN) because it became the biggest source of stress for him. He also mentions that HN `

but it does not work with

`personal_texts = [
    "I love apple pie",
    "My favorite color is fuchsia",
    "My dream is to become a professional dancer",
    "I broke my arm when I was 12",
    "My parents are from Peru",
]

personal_retriever = FAISS.from_texts(personal_texts, embeddings).as_retriever()

`Got invalid return object. Expected markdown code snippet with JSON object, but got:
{
    "destination": "personal",
    "next_inputs": "what are my favorite things?"
}`

[dosubot[bot]]

Hi, @k3nnethfrancis! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you raised is regarding the requirement of a chat model for the MultiRetrievalQAChain in Langchain. There have been discussions on this topic, with some users suggesting solutions like specifying default_chain in MultiRetrievalQAChain.from_retrievers. However, it seems that there are still ongoing issues with parsing errors and invalid return objects when using certain retrievers.

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project!