[Bug]: SWE-Bench official instance image not exist

[FatPigeorz]

Is there an existing issue for the same bug?

• [x] I have checked the existing issues.

Describe the bug and reproduction steps

I try to run openhands on SWE_Bench_Verifed using commands below, but can't pull the instance image successfully, just like issue #7143 .

DEBUG=1 ./evaluation/benchmarks/swe_bench/scripts/run_infer.sh llm.qwq HEAD CodeActAgent 300 30 1 princeton-nlp/SWE-bench_Verified test

I checked the log below

20:05:48 - openhands:WARNING: run_infer.py:109 - Using official SWE-Bench image: sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439:latest
20:05:48 - openhands:INFO: run_infer.py:130 - Using instance container image: docker.io/xingyaoww/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439:latest. Please make sure this image exists. Submit an issue on https://github.com/All-Hands-AI/OpenHands if you run into any issues.

However, the instance image xingyaoww/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439:latest. does not exists. command:

docker pull xingyaoww/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439:latest

output:

Error response from daemon: pull access denied for xingyaoww/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439, repository does not exist or may require 'docker login': denied: requested access to the resource is denied

I searched the image using docker search:

NAME                                                             DESCRIPTION                                     STARS     OFFICIAL
swebench/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439                                                   0         
....
xingyaoww/sweb.eval.x86_64.scikit-learn_s_scikit-learn-13439                                                     0         
....

it seems the right official image should be swebench/sweb.eval.x86_64.scikit-learn_1776_scikit-learn-13439.

The bug seems in the function get_instance_docker_image, the image_name of official image does not return.

def get_instance_docker_image(instance_id: str, official_image: bool = False) -> str:
    if official_image:
        # Official SWE-Bench image
        # swebench/sweb.eval.x86_64.django_1776_django-11333:v1
        repo, name = instance_id.split('__')
        image_name = f'sweb.eval.x86_64.{repo}_1776_{name}:latest'
        logger.warning(f'Using official SWE-Bench image: {image_name}')
+     return ('swebench/' + image_name).lower()
    else:
        # OpenHands version of the image
        image_name = 'sweb.eval.x86_64.' + instance_id
        image_name = image_name.replace(
            '__', '_s_'
        )  # to comply with docker image naming convention
    return (DOCKER_IMAGE_PREFIX.rstrip('/') + '/' + image_name).lower()

OpenHands Installation

Docker command in README

OpenHands Version

main, f51eb93d3ea92231e72247266df6df2d024db826

Operating System

Linux

Logs, Errors, Screenshots, and Additional Context

No response