openai-python
openai-python copied to clipboard
Computer Use Agent -- allow "Action" to be more flexible
Confirm this is a feature request for the Python library and not the underlying OpenAI API.
- [x] This is a feature request for the Python library
Describe the feature or improvement you're requesting
Is there a way to provide custom actions to CUA? Or have custom fields populated in say a returned ActionClick?
For context, we want to figure out what "element" the model is clicking on, and then perform some additional logic apart from simply clicking. CUA also doesn't seem to support structured outputs along with the ResponseComputerToolCall as far as I'm aware, and is also not great at calling other tools while executing a task
Additional context
No response