Agent SELF_ASK_WITH_SEARCH does not work with ChatOpenAI models

[rubensmau]

Got a loop when asking for "What's the best BBQ in Kansas City" . When added to the prompt - "say cannot answer,if you don´t know the answer " did not stop the loop. The loop only stopped after blowing up the context length

On the other hand, if using OpenAI old models, it worked fine . Using SerpAPIWrapper

[oliverjarvis]

I agree, I just stumbled into the very same issue. Also using SerpAPIWrapper.

[djpecot]

I think this may require some tuning to the chat prompt to work effectively

[nkov]

I experience the infinite loop frequently when using gpt-3.5-turbo, but using gpt-4 seems to do a much better job. However, even with gpt-4, I often get an error Could not parse output: Yes., which seems to be the first response from the SELF_ASK_WITH_SEARCH agent when it decides it will proceed with a search. Retrying the same prompt works eventually, so it's kind of hit or miss.

[mykytamykhailenko]

I have a similar issue, but it throws OutputParserException: Could not parse output: No.

[dosubot[bot]]

Hi, @rubensmau! I'm Dosu, and I'm here to help the LangChain team manage their backlog. I wanted to let you know that we are marking this issue as stale.

From what I understand, the issue you reported is related to the Agent SELF_ASK_WITH_SEARCH not working with ChatOpenAI models. It seems that other users, like oliverjarvis and cuongdo, have also encountered this problem. Some users have suggested tuning the chat prompt as a potential solution, while others have found that using gpt-4 instead of gpt-3.5-turbo improves the situation. However, there are still occasional errors reported, such as "Could not parse output: Yes" or "OutputParserException: Could not parse output: No."

Before we close this issue, we wanted to check with you if it is still relevant to the latest version of the LangChain repository. If it is, please let us know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days.

Thank you for your understanding and contribution to the LangChain project! Let us know if you have any further questions or concerns.