OpenHands icon indicating copy to clipboard operation
OpenHands copied to clipboard

[Bug]: CodeActAgent: o1 replies may contain <execute_ipython> in finish action

Open tobitege opened this issue 4 months ago • 5 comments

Is there an existing issue for the same bug?

  • [X] I have checked the troubleshooting document at https://docs.all-hands.dev/modules/usage/troubleshooting
  • [X] I have checked the existing issues.

Describe the bug

Due to how o1-mini works, and probably o1-preview as well, it may return an AgentFinishAction with embedded code. Apparently this is currently not supported in the agent as there is no CodeActActionParserIPythonRunCell triggered.

Example with content manually shortened and line breaks added for readability:

I will create the file o1-results.txt and write the findings from Step 1: 
<execute_ipython>create_file('o1-results.txt') append_file('o1-results.txt', """### Findings for Step 1: Understand the cli.py file

The cli.py file serves as the command-line interface for the OpenHands application. It performs the following functions:
[---content shortened---]
Overall, cli.py orchestrates the setup and execution of the agent in a command-line environment,
facilitating interaction between the user and the underlying agent functionalities. """)
</execute_ipython>

In that step's LLM response log we can see at the bottom:

[---content shortened---]
</execute_ipython>

<finish></finish>

In the main logfile we see the AgentFinishAction (I manually added line breaks here for readability):

10:03:18 - openhands:INFO: agent_controller.py:445 - AgentFinishAction(outputs={}, 
thought='I will create the file `o1-results.txt` and write the findings from Step 1:\n
<execute_ipython>\ncreate_file(\'o1-results.txt\')\n
append_file(\'o1-results.txt\', """### Findings for Step 1: Understand the cli.py file\n
\n
The `cli.py` file serves as the command-line interface for the OpenHands application.
It performs the following functions:\n
[---content shortened---]
Overall, `cli.py` orchestrates the setup and execution of the agent in a command-line environment, 
facilitating interaction between the user and the underlying agent functionalities.\n""")\n
</execute_ipython>', action='finish')

Current OpenHands version

0.9.4 as of 2024-09-24

Installation and Configuration

make build
make run

Model and Agent

  • Model: o1-mini
  • Agent: CodeActAgent

Operating System

WSL

Reproduction Steps

See trajectory: https://www.all-hands.dev/share?share_id=58c3e506b401943c64cc9a2091c7173fe7500c176639ac3fbbdd6659becd1d64

Logs, Errors, Screenshots, and Additional Context

No response

tobitege avatar Sep 24 '24 08:09 tobitege