langchain icon indicating copy to clipboard operation
langchain copied to clipboard

Issues with conversational_chat and LLM chains responding with a multi-line markdown code block

Open surfkansas opened this issue 1 year ago • 2 comments

When using the conversational_chat agent, an issue occurs when the LLM returns a markdown result that has a code block included. My use case is an agent that reads from an internal database using an agent to get information that will be used to build a block of code. The agent correctly uses the tools, but fails to return an answer.

The exhibited behavior is that the triple apostrophe used to offset the code block interferes with the json in a code block that the agent is expecting. It tries to parse the document, but stops when it finds the second triple apostrophe that starts the code block. This is because we end up with nested code blocks (the outer code block being the agent response JSON in a code block)

surfkansas avatar Mar 31 '23 18:03 surfkansas

Same! Major issue

Kav-K avatar Apr 24 '23 15:04 Kav-K

Same problem if you just try to run the router-chain example (using AzureOpenAI): https://python.langchain.com/en/latest/modules/chains/generic/router.html

krlng avatar May 22 '23 09:05 krlng

One solution would be to modify this prompt so that all of the examples use quadruple backticks.

That should be transparent to any existing use case, yet preserve the use case you describe where the response contains a nested code block.

Opened a PR for this: https://github.com/langchain-ai/langchain/pull/9781

wnmurphy avatar Aug 26 '23 00:08 wnmurphy

Hi, @surfkansas! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you reported is related to the conversational_chat agent in the langchain repository. It seems that the agent fails to return an answer when the LLM returns a markdown result with a code block. This has been identified as a major problem, and one proposed solution is to modify the prompt so that all examples use quadruple backticks. This solution aims to address the nested code block problem, while still preserving existing use cases. I see that a pull request has been opened for this solution.

Before we proceed, we would like to confirm if this issue is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on this issue. Otherwise, feel free to close the issue yourself, or the issue will be automatically closed in 7 days.

Thank you for your understanding and cooperation. Let us know if you have any further questions or concerns!

Best regards, Dosu

dosubot[bot] avatar Nov 25 '23 16:11 dosubot[bot]