Using with WindowsAgentArena

[sachit-menon]

Hi, thanks for the great work! I saw that the S2 model shows great results on WindowsAgentArena as well as OSWorld and that the Changelog references gui-agents being easy to use for Mac, OSWorld, WindowsAgentArena. I was wondering how to use it with WindowsAgentArena. I have OSWorld set up but haven't tried adding this agent to it; if I understand correctly, it's mostly a matter of copying the run files from this repo to that folder. Is it similar for WindowsAgentArena? Would love any details to reproduce the evaluation.

[alckasoc]

Hi @sachit-menon , sorry for the late response!

The WindowsAgentArena setup requires a bit more effort, but I've attached the setup guide below. If you'd like, we also have the VM image, but it has not been reset from the last example (though it is setup).

WindowsAgentArena Setup.pdf

A bit of info on the setup guide:

• you can disable perplexica if there are docker issues
• installing the LibreOffice apps is a must to get the full benchmark
• closing the initial pop-ups isn't required but should be done to simplify any unexpected initial noise and allow for the initialization of tasks to be done correctly (otherwise your agent may just be dealing with the pop-ups and the initial state may not be on the correct screen)
• set_cell_values is an action to set spreadsheet values in LibreOffice Calc and most of the setup nuance is on this (if you'd like, you can disable this action and run without it)

Hope this helps!