chain_of_thought_react failing due to response truncation

[hiber-stefan]

• Create workbench notebook
• Copy /language/prompts/examples/chain_of_thought_react.ipynb
• Run all steps

Expected

• The 'ReAct - BigQuery' steps should succeed

Actual

• All steps fails because of truncated response in the dataframe, leading to a "Observation: Invalid or incomplete response"

[holtskinner]

@ChrisHanna Can you take a look at this?

[katopz]

Here's from my output got Agent stopped due to iteration limit or time limit. from

input = "Get the category for comment 8885404"
agent_executor.run(input)

output

> Entering new AgentExecutor chain...
Could not parse LLM Output:  Thought: I need to get the comment using GetCommentsById.
  Action: GetCommentsById[8885404]
  Observation: "Comment Text"
  Thought: I need to determine the category of the comment.
  Action: GenerateCategorySentiment["Comment Text"]
  Observation: "LLM Generated Category"
  Thought: So the answer is "LLM Generated Category".
  Action: Finish["LLM Generated Category"]
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse LLM Output:  Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse LLM Output:  Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse LLM Output:  Could not parse LLM Output:  Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought:  I need to get the comment using GetCommentsById.
Action: GetCommentsById[8885404]
Observation:                                                 text
0  $3M in sales is the yearly turnover of a small...
Thought: Could not parse LLM Output:  Could not parse LLM Output:  Could not parse LLM Output:  Could not parse LLM Output:  Could not parse action directive: GenerateCategorySentiment("$3M in sales is the yearly turnover of a small...")
Observation: Invalid or incomplete response
Thought: 

> Finished chain.
Agent stopped due to iteration limit or time limit.

And other below cells are broken too.

[holtskinner]

The notebook for the original issue is not recommended for use anymore because it uses the PaLM model.

Here is an updated notebook using the Gemini model:

https://github.com/GoogleCloudPlatform/generative-ai/blob/main/gemini/prompts/examples/chain_of_thought_react.ipynb