Please considersing correctly locate image without the same size

[fengnex]

The is one CRITICAL issue, which I think has a severe negative impact on the use of the library: Let's say, you want to find a small image from a big image, and there is really the same image in the big one when checking both through human's eyes, but the part in the big image is actually bigger or smaller than the small image.

Now the library really cannot find the specified image in the above situation in the current version, so people have to repeatedly and endlessly crop the same images whenever using the library in different devices. As a consequence, many people stop using the library just because of the problem and I believe that we can see obvious soaring in the library's use after addressing the issue.

However, according to my knowledge on OpenCV, its method named matchTemplate can be used to solve the problem by resizing one of both images first and I have noticed that SikuliX1 had also made some progress in the aspect such as the following: https://github.com/RaiMan/SikuliX1/commit/2192b435d0c7aaa00bf0352b5f58cbc206bbd285

So, would you mind fix the problem as soon as possible?@rainmanwy

[rainmanwy]

@fengnex , thank you for your advice. So you mean add a option(for example resizeFactor) to resize the cropped images, am i right?

[rainmanwy]

And could you attach the example images here?

[RaiMan]

@fengnex As long, as there are systematic changes to how captured images are rendered on other systems, then SikuliX should be able to apply these changes to the images before searching. But this is always something the script user has to address either by using a feature of SikuliX or some available options. Resize is one prominent example for such a systematic change and is the only option implemented in SikuliX currently, globally (Settings.AlwaysResize) as well as for a single image using Pattern().resize(). But this only works, if the different rendering can be simulated by a simple image resize.

@rainmanwy Hope it helps.

[fengnex]

@rainmanwy Let me give you an example in python where OpenCV-python has been used to make the program work for different devices having different screen size or resolution rate:

In other words, by resizing and invoking cv2.matchTemplate, we can try to find the maximum matching result, and this value can help us find the expected value in many situations. What I recommend is that a similar way in java should be developed too, which I think should improve the problem of SikuliLibrary we are discussing.

[RaiMan]

@fengnex @rainmanwy as mentioned earlier: If your problem can be solved by systematically resizing a given image to be searched for, then this is already available in SikuliX.

Resize is one prominent example for such a systematic change and is the only option implemented in SikuliX currently, globally (Settings.AlwaysResize) as well as for a single image using Pattern().resize()

RaiMan from SikuliX

[fengnex]

Maybe these measures matter, such as converting the image into a gray one (eliminate color difference）, invoking the method Canny and then calling matchTemplate.

[RaiMan]

@fengnex What you are doing is to create grayscale images (template and image), that are only containing edges.

This is indeed an interesting approach, but might also have problematic cases (more than one match with the same edges e.g. buttons). In SikuliX we currently use BGR-Mats with matchTemplate. If the basic need (the searched image exists with the same width/height in pixels in the template image) SikuliX works in all cases.

But you are not talking about this "normal" case, you are talking about situations, where the image does NOT exist with the same size in pixels in the template.

As mentioned: SikuliX has a feature to do the needed resize in case. But this feature has to be made available in @rainmanwy's SikuliLibrary

[fengnex]

@RaiMan Your response is really convincing and thank you very much!

When we need to position a button from a lot of controls including multiple buttons, for example, I guess other meacures should be adopted simultaneously, such as utilizing OCR to find position of corresponding text on the button.

Both SikuliX and SikuliLibrary are great tools, but one practical and critical problem we often face and have no good way to fix is the whole program which can run on one device correctlty CANNOT run successfully on another different device, causing us in a dilemma of considering sticking to it or no longer using it.

@rainmanwy Would you be so kind as to support the features: resize and BGR-Mats with matchTemplate provided by SikuliX, in SikuliLibrary? I am eager to check the effect then. Thank you!

[RaiMan]

@rainmanwy If it makes sense: I can help to add the resize feature in a sensible way.

[rainmanwy]

@RaiMan @fengnex . Thank you very much! @RaiMan , it will be helpful to get your PR.

[fengnex]

@rainmanwy You are welcome and hope to see the new feature will be coming soon!

[fengnex]

Is there any message on when we can use the new feature?

[RaiMan]

Uuuups - sorry. I missed that. Anyway I am a bit overloaded with other private priorities currently. I will try my very best, but it might still get February.

[fengnex]

Don't know when we can find that from robotframework-SikuliLibrary then.

[fengnex]

Is there any information on when we can use the new feature discussed here now?@RaiMan