SikuliX1
SikuliX1 copied to clipboard
RaiMan
Detect that a match is differently coloured
(base for this issue: https://github.com/RaiMan/SikuliNG/issues/3)
The most prominent example are GUI buttons, that signal there state by different colors.
Here an example (thanks to @seltix5):
# the image used for search:
button = 
# the button as shown on screen (coloured)
img1 =
# the button as shown on screen (greyed)
img2 =
The match score is evaluated with button in button, img1 and img2:
(since I do not have access to the real screen content, I use the provided images)
With 2.1.0:
print "button:", Image(button).find(button)
print "img1:", Image(img1).find(button)
print "img2:", Image(img2).find(button)
# prints:
button: M[0,0 162x21 S(100.00) (24, 21, 1, 2)]
img1: M[46,18 162x21 S(99.98) (4, 3, 0, 1)]
img2: M[38,12 162x21 S(97.39) (1, 1, 0, 0)]
(the last bracketed values are timing values in milliseconds)
With 2.0.3:
finder = Finder(Image.create(button))
finder.find(button)
print "button:", finder.next().getScore()
finder = Finder(Image.create(img1))
finder.find(button)
print "img1:", finder.next().getScore()
finder = Finder(Image.create(img2))
finder.find(button)
print "img2:", finder.next().getScore()
# prints:
button: 0.999999344349
img1: 0.999807238579
img2: 0.973917245865
... which shows, that the score between coloured and greyed differs by >0.02.
The problem/challenge: In this case with only one shot, the score alone says nothing about, how the match is coloured. One needs additional information to decide (in this case that the score is less than 0.99 but greater than 0.97).
A workaround is to use findBest() or findAny(), but then you would need 2 shots: coloured and greyed.
Hello, Since the current behavior is probably desired for most case, any update related to this subject probably should be optional by new param or dedicated methods. Currently, how mutch the color weight in the final score? Does it compare the average or most common colors?
Lowering the similitude param is probably not a good option since it will decresce accuracy. Is it not possible to add new method to increase the color comparison weight?
PS: Currently the sikuli ide does not show the scores of the matches found, it wold be super helfull for development to have that information plus the total of matches found. Edited : I'm currently develping on Java but I still use sikuli ide to capture the shots since the tool to previous the matches found depending on the similaritude chosen is crucial for debugging and find the optimal/maximum similaritude value.
any update related to this subject probably should be optional by new param or dedicated methods
agreed. It will anyways always be a 2-step process:
- find the match
- make additional evaluations (e.g. compare mean-color)
I think the Pattern class is the place to say: respect the mean color. With this approach we do not need extra functions. The search features will return matches with respect to the Pattern attributes.
Lowering the similitude param is probably not a good option
This is not really true. If you have a good shot, you should get a score above 0.9 or even 0.95 - in most cases 0.99+. As shown above in my example, the difference between different color cases is less than 0.05. So if you now work with the standard 0.7 expected score, now saying, I expect a score of 0.9 to match both cases and then in a second step decide for the exact match, is absolutely ok.
Currently the sikuli ide does not show the scores of the matches found
In the IDE Preview window you have the middle tab (Matching Preview), where you can evaluate the match score with the slider at the bottom. The number of matches is not shown as a number - you have to count the coloured squares.
I think the Pattern class is the place to say: respect the mean color. With this approach we do not need extra functions. The search features will return matches with respect to the Pattern attributes.
Agree, the simpleer implementation wold be a method in the pattern class to control the color accuracy. I wold also create methods in the match class to retrieve the final, the image and color match scores indendently if possible, to make the adicional evaluations when needed. This way I could search for a normal pattern and only after check the color match score.
This is not really true. If you have a good shot, you should get a score above 0.9 or even 0.95 - in most cases 0.99+. As shown above in my example, the difference between different color cases is less than 0.05. So if you now work with the standard 0.7 expected score, now saying, I expect a score of 0.9 to match both cases and then in a second step decide for the exact match, is absolutely ok.
True but we will need to do later evaluations any way meaning changing the similaritute score will not help. I could set the pattern similaritute to something like .98 or .99 to be sure to get only the not activate button but with this hight values some noise on the source will probably cause a FindFail.
In the IDE Preview window you have the middle tab (Matching Preview), where you can evaluate the match score with the slider at the bottom. The number of matches is not shown as a number - you have to count the coloured squares.
Yes, I was talking about that tab, but I dont see any score... I see parentices in front of the slider but they are empty
Additional functions in Pattern and Match class
Agreed. Sorry, for not having been complete ;-)
High expected score values
As mentioned, I was talking about good captures. Good captures have as little background as possible and in best case no areas that might become noisy. So if you have 2 possible cases, then take the lower of the 2 possible values (- 0.05) and you are on the safe side. If there is no risk for false positives, you might of course do not use Pattern.similar() at all.
Preview window
On Windows I have at the right side of the slider (0.70) ( ) and the shown value changes according to the slider position.
... but anyways, the Preview feature will be revised in 2.1.0 and has new features. I will keep an eye on it ;-)
On Windows I have at the right side of the slider (0.70) ( ) and the shown value changes according to the slider position.
yes the selected values changes according to the slider, but I was talking about showing the matches scores. if I understood correctly the background of the highlight changes according to the match score, but the value itself is not visible.
Uuups, thanks for your patience - now I got it.
You want to have a score value with each coloured rectangle. Ok, on the list now.
no problem, glad to help. It wold be helpful for faster analysis during development =)
Hello, I am interested in this enhancement. I am using your library for automated testing and the colour is often a relevant factor. Is this feature in progress by anyone? Perhaps I could try to implement it and submit a pull request if I get some time in the next weeks.
@MrMisterG No progress from my side.
Before investing any effort into code evaluation and coding you should:
- provide one or more examples about
colour is often a relevant factor - a proposal how an enhancement should look like with respect to features and API (so we can discuss it)
If you want to dive into the details, then the reference code base currently is the branch release_2.0.x
(neither master(2.1.0) nor SikuliNG are currently actively developed)