Running SWE-agent with SWE-bench Dataset

[ramsey-coding]

SWE-agent Team!

Great work, congrats.

I am attempting to run the SWE-agent on the SWE-bench dataset and have encountered some challenges in understanding the process. I see some scripts in the repository at this link, which seem relevant to what I am trying to achieve.

However, the documentation does not outline the steps for executing these scripts, collecting patches, running them against the test suites, and then gathering the results. Could you please provide help me?

• Executing Scripts: how to run the scripts located in the scripts directory.
• Collecting Patches: how to collect patches generated by SWE-agent.
• Running Test Suites: run these patches against test suites within the SWE-bench dataset.
• Gathering Results: how to best collect and interpret the results from the test runs.

Thanks!

[klieret]

Duplicate of #67

Edit: Actually let me leave this one open and close the other one because of the more concise questions here ;)

I'm unfortunately relatively new to the team and not too familiar with SWE-bench, but I hope to get back to you soon (or pull in someone more knowledgeable than me).

[ramsey-coding]

@klieret really appreciate for looking into it. I would really appreciate your help with this issue. At least how to run it against the SWE-Bench.

[ofirpress]

After we post the preprint @john-b-yang and @carlosejimenez will provide all the details for running SWE-agent on SWE-bench.