Improvements to ConvoOutputParser and MessageHistory

[Cobryis]

• Updated ConvoOutputParser to more robustly handle poorly formatted response scenarios from an LLM like GPT-3.5-Turbo.

• Added a ActionBasedChatMessageHistory as an alternative to ChatMessageHistory to avoid prompt contamination caused by LLM reading its previous messages without proper formatting and then not correctly formatting future responses.

[Cobryis]

Just wanted to add a comment that for ConvoOutputParser I did consider raising an exception if there was a json object in the response but it was missing the required fields. I opted not to do this because in general with this change I'm expecting incorrectly formatted responses from the LLM to be Final Answers. However, if the agent were able to receive and understand exceptions so it could correct itself, I might consider it better to raise the exception (though this would introduce higher token usage which I'm currently trying to avoid).

[skcoirz]

Hi @Cobryis , I like your idea of customizing json decoder! Left some small comments. Will leave for the maintainers to decide. :)

btw, it looks like we don’t have test case for ConvoOutputParser yet. Will it be a good idea to add a test for it? Thanks!

[Cobryis]

p

Yeah I think a test for it would be good. I'm currently working on a real application using this stuff so that's how I'm testing at the moment. When I have more free time I'd like to build more tests (or a notebook) mimicking what I have so that this stuff is less fragile.

[Ryangr0]

Author

Really cool. I'm doing the same and running into the same kind of error I think.

File "/app/output_parsers/CustomOutputParser.py", line 21, in parse
    raise ValueError(f"Could not parse LLM output: `{llm_output}`")
ValueError: Could not parse LLM output: `
Thought: Scan the repository and make a detailed analysis of it.
Action:
{
  "action": "TODO",
  "action_input": "Scan the repository you're in and make a detailed analysis of it. Then put it in a file called 'helloworld.md'"
}
`

I love this community is so active right now.

[hwchase17]

p

Yeah I think a test for it would be good. I'm currently working on a real application using this stuff so that's how I'm testing at the moment. When I have more free time I'd like to build more tests (or a notebook) mimicking what I have so that this stuff is less fragile.

echoing this, a small test suite would be AWESOME here

love this effort tho - thanks for improving this

[clockcoinG1]

I added a very powerful bash agent, let me know if you are interested. I also allow it to make python / ruby/ any code available from thee env

[arnabbiswas1]

@hwchase17 Are you planning to merge this PR? I understand that you are working for a test suite.

[rickbraddy-pharma]

Is this fix available ?

[0xrjman]

hey, any update on this PR?

[rickbraddy-pharma]

It was abandoned

On Aug 31, 2023, at 3:38 PM, 0xrjman @.***> wrote:

 Caution: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe. When in doubt, contact your IT Department

hey, any update on this PR?

— Reply to this email directly, view it on GitHubhttps://github.com/langchain-ai/langchain/pull/3480#issuecomment-1701748501, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AYDQ4EVABN4KKNGYHRUGJZTXYDY2XANCNFSM6AAAAAAXKL7RF4. You are receiving this because you commented.Message ID: @.***>

[leo-gan]

@Cobryis Hi , could you, please, resolve the merging issues and address the last comments (if needed)? After that, ping me and I push this PR for the review. Thanks!

If this PR is not needed anymore, could you, please, let me know?

[Cobryis]

@Cobryis Hi , could you, please, resolve the merging issues and address the last comments (if needed)? After that, ping me and I push this PR for the review. Thanks!

If this PR is not needed anymore, could you, please, let me know?

I'm not sure if it's not needed, but I'm not using it. It seems like some people would like this, though they're free to fork my change and get it to marge with the latest in langchain. I'm not using langchain anymore so really not sure what the code looks like now.

[leo-gan]

@Cobryis Thanks for feedback! Thanks for your thoughts and effort! Really appreciated!