Computer use

[avaer]

What do you want to see?

Allow agents to run on the desktop using electron. Automation courtesy of @nut-tree-fork/nut-js.

This is to support a live desktop control experience similar to the popular https://github.com/corbt/agent.exe

Docs: https://docs.anthropic.com/en/docs/build-with-claude/computer-use

Example from Anthropic: https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo

To make this work, we need to support a pipeline for Anthropic/Claude inference. We are already proxying these calls in the ai proxy.

---

The available tools are currently (per https://docs.anthropic.com/en/docs/build-with-claude/computer-use):

{ "type": "computer_20241022", "name": "computer" }
{ "type": "text_editor_20241022", "name": "str_replace_editor" }
{ "type": "bash_20241022", "name": "bash" }

Examples of these being used is here: https://github.com/anthropics/anthropic-quickstarts/tree/main/computer-use-demo/computer_use_demo/tools

Link to original discussion

No response