openai-python icon indicating copy to clipboard operation
openai-python copied to clipboard

Computer Use Agent -- allow "Action" to be more flexible

Open rahuls1996 opened this issue 9 months ago • 0 comments

Confirm this is a feature request for the Python library and not the underlying OpenAI API.

  • [x] This is a feature request for the Python library

Describe the feature or improvement you're requesting

Is there a way to provide custom actions to CUA? Or have custom fields populated in say a returned ActionClick?

For context, we want to figure out what "element" the model is clicking on, and then perform some additional logic apart from simply clicking. CUA also doesn't seem to support structured outputs along with the ResponseComputerToolCall as far as I'm aware, and is also not great at calling other tools while executing a task

Additional context

No response

rahuls1996 avatar Mar 13 '25 03:03 rahuls1996