SWE-bench

SWE-bench copied to clipboard

Enhancements to run_evaluation.py: This pull request adds an optional command-line argument `--path_conda` to the run_evaluation.py script. The main functionality enhancement is the ability to specify a custom path to a...

Jonty800

I would be extremely grateful if you could share the log files from the baseline model testing process discussed in your study. Access to these logs, specifically those with names...

yczhou001

What are expected to submit for the leaderboard integration?

2

Check https://www.swebench.com and found: 

zhimin-z

It would be nice to have a "Experiments" directory for reproducible research via MLflow or a similar tool.

moresearch

May I ask where can I download the generated results from Claude and GPTs?

6

I have read your paper and really like this work. Could I ask where can I download the generated results from Claude and GPTs? These results are beneficial to our...

itaowei

[HuggingFace Endpoints](https://huggingface.co/inference-endpoints/dedicated) seems to be an easier way to run SWE-Llama models on the cloud. Tips for coding this feature?

moresearch

Tests that should fail don't fail.

3

Thank you for providing a great data set. I have tested it in my environment and confirmed that the fail to pass test does not fail in the following instances,...

t-kurabayashi

Issues in the test case parsing logic from the logs

4

Firstly, thanks for this useful dataset. Some of the pytest logs have a format such as this: ``` PASSED sklearn/feature_extraction/tests/test_text.py::test_callable_analyzer_error[file-AttributeError-'str' object has no attribute 'read'-CountVectorizer] PASSED sklearn/feature_extraction/tests/test_text.py::test_callable_analyzer_error[file-AttributeError-'str' object has no...

anmolagarwal999

logs are unusable with multiple test instances

4

`harness/run_evaluation.py` takes a `--log_dir` argument but if there are multiple predictions for a single model and test in the `--predictions_path` file they all write to a single file in the...

JasonGross

sometimes gold_patch cannot pass the test

6

This is a very challenging benchmark, I have learned a lot from it. Thank you for the effort you have put into this. I tested using the swe-llama13b you provided...

LuoKaiGSW