langchain
langchain copied to clipboard
RetrievalQA & RetrievalQAWithSourcesChain chains cannot be serialized/saved or loaded
VectorDBQA
is being deprecated in favour of RetrievalQA
& similarly, VectorDBQAWithSourcesChain
is being deprecated for RetrievalQAWithSourcesChain
.
Currently, VectorDBQA
& VectorDBQAWithSourcesChain
chains can be serialized using vec_chain.save(...)
because they have _chain_type
property - https://github.com/hwchase17/langchain/blob/3bd5a99b835fa320d02aa733cb0c0bc4a87724fa/langchain/chains/qa_with_sources/vector_db.py#L67
However, RetrievalQA
& RetrievalQAWithSourcesChain
do not have that property and raise the following error when trying to save with ret_chain.save("ret_chain.yaml")
:
File [~/.pyenv/versions/3.8.15/envs/internalqa/lib/python3.8/site-packages/langchain/chains/base.py:45](https://file+.vscode-resource.vscode-cdn.net/Users/smohammed/Development/hackweek-internalqa/~/.pyenv/versions/3.8.15/envs/internalqa/lib/python3.8/site-packages/langchain/chains/base.py:45), in Chain._chain_type(self)
[43](file:///Users/smohammed/.pyenv/versions/3.8.15/envs/internalqa/lib/python3.8/site-packages/langchain/chains/base.py?line=42) @property
[44](file:///Users/smohammed/.pyenv/versions/3.8.15/envs/internalqa/lib/python3.8/site-packages/langchain/chains/base.py?line=43) def _chain_type(self) -> str:
---> [45](file:///Users/smohammed/.pyenv/versions/3.8.15/envs/internalqa/lib/python3.8/site-packages/langchain/chains/base.py?line=44) raise NotImplementedError("Saving not supported for this chain type.")
NotImplementedError: Saving not supported for this chain type.
There isn't any functions to support loading RetrievalQA either, unlike the VectorQA counterparts: https://github.com/hwchase17/langchain/blob/3bd5a99b835fa320d02aa733cb0c0bc4a87724fa/langchain/chains/loading.py#L313-L356
Facing the same issue
Facing similar issue for retrivalQA, with elasticsearch as source. Any idea how to resolve this?
Same issue with pinecone as a source
ditto. I can get both a conversational chain and sources back from the pinecone with bit not the retrieval counterpart. sources are being sent from the db to the agnet and sent to the llm but not showing up in the final answers (maybe a separate problem)
Just want to mention, this was working fine in previous versions, but I'm not sure when exactly it got broken.
same issue for RetrievalQA!
@adamyodinsky are you able to point to a specific version for which this worked?
@adamyodinsky are you able to point to a specific version for which this worked?
@dsaks9 No, I'm sorry I can't.
I'm on Langchain v0.0.219 and still running into this issue with RetrievalQAWithSourcesChain where I'm getting an error that Saving not supported for this chain type.
Note that I have RetrievalQAWithSourcesChain defined as a tool going into another agent. (This works fine on its own with a qa_chain(query))
qa_chain =RetrievalQAWithSourcesChain.from_chain_type(llm=llm, chain_type="stuff", retriever=pinecone_retriever.as_retriever(), verbose=True, reduce_k_below_max_tokens=True)
Here I turn it into a tool
from langchain.agents import Tool
tools = [
Tool(
name = "Get Knowledgebase Data",
func=qa_chain,
description="useful for when you need to answer questions about the.... knowledgebase about content ...."
)]
agent_aggregator = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
Doing a query on agent_aggregator has the following error. I did remove this specific tool and everything works fine, so it's isolated to RetrievalQAWithSourcesChain being a tool in this other agent.
> Entering new chain...
I should check the Nanome knowledgebase data to find information about supported VR headsets.
Action: Get Nanome Knowledgebase Data
Action Input: Supported VR headsets
---------------------------------------------------------------------------
NotImplementedError Traceback (most recent call last)
[<ipython-input-19-52f895e71d87>](https://localhost:8080/#) in <cell line: 1>()
----> 1 agent_aggregator("query question about the knowledge_base input")
13 frames
[/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py](https://localhost:8080/#) in __call__(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
164 except (KeyboardInterrupt, Exception) as e:
165 run_manager.on_chain_error(e)
--> 166 raise e
167 run_manager.on_chain_end(outputs)
168 final_outputs: Dict[str, Any] = self.prep_outputs(
[/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py](https://localhost:8080/#) in __call__(self, inputs, return_only_outputs, callbacks, tags, include_run_info)
158 try:
159 outputs = (
--> 160 self._call(inputs, run_manager=run_manager)
161 if new_arg_supported
162 else self._call(inputs)
[/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py](https://localhost:8080/#) in _call(self, inputs, run_manager)
985 # We now enter the agent loop (until it returns something).
986 while self._should_continue(iterations, time_elapsed):
--> 987 next_step_output = self._take_next_step(
988 name_to_tool_map,
989 color_mapping,
[/usr/local/lib/python3.10/dist-packages/langchain/agents/agent.py](https://localhost:8080/#) in _take_next_step(self, name_to_tool_map, color_mapping, inputs, intermediate_steps, run_manager)
848 tool_run_kwargs["llm_prefix"] = ""
849 # We then call the tool on the tool input to get an observation
--> 850 observation = tool.run(
851 agent_action.tool_input,
852 verbose=self.verbose,
[/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py](https://localhost:8080/#) in run(self, tool_input, verbose, start_color, color, callbacks, **kwargs)
297 except (Exception, KeyboardInterrupt) as e:
298 run_manager.on_tool_error(e)
--> 299 raise e
300 else:
301 run_manager.on_tool_end(
[/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py](https://localhost:8080/#) in run(self, tool_input, verbose, start_color, color, callbacks, **kwargs)
269 tool_args, tool_kwargs = self._to_args_and_kwargs(parsed_input)
270 observation = (
--> 271 self._run(*tool_args, run_manager=run_manager, **tool_kwargs)
272 if new_arg_supported
273 else self._run(*tool_args, **tool_kwargs)
[/usr/local/lib/python3.10/dist-packages/langchain/tools/base.py](https://localhost:8080/#) in _run(self, run_manager, *args, **kwargs)
410 ) -> Any:
411 """Use the tool."""
--> 412 new_argument_supported = signature(self.func).parameters.get("callbacks")
413 return (
414 self.func(
[/usr/lib/python3.10/inspect.py](https://localhost:8080/#) in signature(obj, follow_wrapped, globals, locals, eval_str)
3252 def signature(obj, *, follow_wrapped=True, globals=None, locals=None, eval_str=False):
3253 """Get a signature object for the passed callable."""
-> 3254 return Signature.from_callable(obj, follow_wrapped=follow_wrapped,
3255 globals=globals, locals=locals, eval_str=eval_str)
3256
[/usr/lib/python3.10/inspect.py](https://localhost:8080/#) in from_callable(cls, obj, follow_wrapped, globals, locals, eval_str)
3000 follow_wrapped=True, globals=None, locals=None, eval_str=False):
3001 """Constructs Signature for the given callable object."""
-> 3002 return _signature_from_callable(obj, sigcls=cls,
3003 follow_wrapper_chains=follow_wrapped,
3004 globals=globals, locals=locals, eval_str=eval_str)
[/usr/lib/python3.10/inspect.py](https://localhost:8080/#) in _signature_from_callable(obj, follow_wrapper_chains, skip_bound_arg, globals, locals, eval_str, sigcls)
2465 globals=globals, locals=locals, eval_str=eval_str)
2466
-> 2467 if _signature_is_builtin(obj):
2468 return _signature_from_builtin(sigcls, obj,
2469 skip_bound_arg=skip_bound_arg)
[/usr/lib/python3.10/inspect.py](https://localhost:8080/#) in _signature_is_builtin(obj)
1992 # Can't test 'isinstance(type)' here, as it would
1993 # also be True for regular python classes
-> 1994 obj in (type, object))
1995
1996
/usr/local/lib/python3.10/dist-packages/pydantic/main.cpython-310-x86_64-linux-gnu.so in pydantic.main.BaseModel.__eq__()
[/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py](https://localhost:8080/#) in dict(self, **kwargs)
340 raise ValueError("Saving of memory is not yet supported.")
341 _dict = super().dict()
--> 342 _dict["_type"] = self._chain_type
343 return _dict
344
[/usr/local/lib/python3.10/dist-packages/langchain/chains/base.py](https://localhost:8080/#) in _chain_type(self)
63 @property
64 def _chain_type(self) -> str:
---> 65 raise NotImplementedError("Saving not supported for this chain type.")
66
67 @root_validator()
NotImplementedError: Saving not supported for this chain type.
Major apologies if this is the wrong issue thread, I'm pretty sure it's related but can move it somewhere more appropriate if needed.
Any thought on what could be happening?
@shessenauer
Having the exact same problem!
Here's my stacktrace:
`
Entering new chain... I need to find information about electric vehicles Action: Findall QA with Sources System Action Input: Trends in electric vehicles I need to find information about electric vehicles Action: Findall QA with Sources System Action Input: Trends in electric vehicles2023-07-13 12:25:48.792 Uncaught app exception Traceback (most recent call last): File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/streamlit/runtime/scriptrunner/script_runner.py", line 552, in _run_script exec(code, module.dict) File "/home/fabmeyer/Dev/Python/personal-investment-assistant/personal_assistant.py", line 272, in
response = st.session_state['agent']({'input': user_input}, return_only_outputs=True) File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/chains/base.py", line 243, in call raise e File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/chains/base.py", line 237, in call self._call(inputs, run_manager=run_manager) File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/agents/agent.py", line 987, in _call next_step_output = self._take_next_step( File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/agents/agent.py", line 850, in _take_next_step observation = tool.run( File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/tools/base.py", line 320, in run raise e File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/tools/base.py", line 292, in run self._run(*tool_args, run_manager=run_manager, **tool_kwargs) File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/tools/base.py", line 442, in _run new_argument_supported = signature(self.func).parameters.get("callbacks") File "/home/fabmeyer/.pyenv/versions/3.9.0/lib/python3.9/inspect.py", line 3118, in signature return Signature.from_callable(obj, follow_wrapped=follow_wrapped) File "/home/fabmeyer/.pyenv/versions/3.9.0/lib/python3.9/inspect.py", line 2867, in from_callable return _signature_from_callable(obj, sigcls=cls, File "/home/fabmeyer/.pyenv/versions/3.9.0/lib/python3.9/inspect.py", line 2321, in _signature_from_callable if _signature_is_builtin(obj): File "/home/fabmeyer/.pyenv/versions/3.9.0/lib/python3.9/inspect.py", line 1861, in _signature_is_builtin obj in (type, object)) File "pydantic/main.py", line 911, in pydantic.main.BaseModel.eq File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/chains/base.py", line 556, in dict _dict["_type"] = self._chain_type File "/home/fabmeyer/.local/share/virtualenvs/personal-investment-assistant-Y3o7tQcK/lib/python3.9/site-packages/langchain/chains/base.py", line 94, in _chain_type raise NotImplementedError("Saving not supported for this chain type.") NotImplementedError: Saving not supported for this chain type. `
When using the Retrieval QA From Sources chain everything works fine. However I have to combine it with other tools into an agent.
See this other issue: https://github.com/hwchase17/langchain/issues/1844 however it didn't helped me.
I am on LangChain 0.0.231
Hi @fabmeyer @shessenauer I am also trying with the same approach, do you have any resolutions on the issues since your last attempt?
@shessenauer @fabmeyer @mikexd-dev I have the same problem, were you able to resolve this?
@mikexd-dev @skeretna No I was not able to solve the problem. However I decided to add a UI element to select the model/tool and "hardcode" it rather than using an agent to select the appropriate model/tool.
This worked for me when I called the chain using lambda instead of directly, no idea why
So, Change from this:
from langchain.agents import Tool
tools = [
Tool(
name = "Get Knowledgebase Data",
func=qa_chain,
description="useful for when you need to answer questions about the.... knowledgebase about content ...."
)]
agent_aggregator = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
to this
from langchain.agents import Tool
tools = [
Tool(
name = "Get Knowledgebase Data",
func = lambda q: qa_chain(q), #Note the lambda
description="useful for when you need to answer questions about the.... knowledgebase about content ...."
)]
agent_aggregator = initialize_agent(tools, llm, agent=AgentType.ZERO_SHOT_REACT_DESCRIPTION, verbose=True)
@skeretna seems it works with this change, but need modification, add custom output parser can help I think or for example use the lambda q: qa_chain(q)['answer'] and if you need to keep somewhere the info about source, just create a function with this logic and put it in the tool as function