(computer vision, teserract/OpenCV lib for different server issue) Two issue of CN REGULAR

[TW-Raino]

1. The script would stop after Refill. --> Add command [NotJPserverForStaminaRefillExtraClick = 1] can fix this issue.

2. The Strategy of select card didn't work well. --> I think the function checkCardAffin and checkCardType couldn't recognize card correctly.

[29988122]

--> Add command [NotJPserverForStaminaRefillExtraClick = 1] can fix this issue. 看來國服改版行為跟上其他服了，確實這條該拿掉

--> I think the function checkCardAffin and checkCardType couldn't recognize card correctly. 難辦，國服的字體三天一小改五天一大改，挺難跟上.... 您有空的話幫我多拍幾張選卡畫面，越多越好，沒有經過縮圖的螢幕截圖，要Buster, Arts, Quick, Resist Buster, Resist Arts, Resist Quick, Weak Buster, Weak Arts, Weak Resist都有的，從者不拘

我再來看看要怎麼修復，不然我國服沒有帳號很難測... QQ

[TW-Raino]

Hi, Jerry :

我上傳圖片到Google Drive: https://drive.google.com/file/d/1db8iuQQ6bBKPEWaMrHT4sOrA9Rwq-WVB/view?usp=sharing

有缺圖在反應給我!

Jerry Ho [email protected] 於 2018年9月30日 週日 上午11:43寫道：

--> Add command [NotJPserverForStaminaRefillExtraClick = 1] can fix this issue. 看來國服改版行為跟上其他服了，確實這條該拿掉

--> I think the function checkCardAffin and checkCardType couldn't recognize card correctly. 難辦，國服的字體三天一小改五天一大改，挺難跟上.... 您有空的話幫我多拍幾張選卡畫面，越多越好，沒有經過縮圖的螢幕截圖，要Buster, Arts, Quick, Resist Buster, Resist Arts, Resist Quick, Weak Buster, Weak Arts, Weak Resist都有的，從者不拘

我再來看看要怎麼修復，不然我國服沒有帳號很難測... QQ

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/29988122/Fate-Grand-Order_Lua/issues/75#issuecomment-425692001, or mute the thread https://github.com/notifications/unsubscribe-auth/Apf6VfSqKoRiMr2qADoOsdLLFF0OjePVks5ugD3FgaJpZM4XAm6a .

[29988122]

我不確定能不能行，因為Ankulua的天限，還有這字實在太小了，辨識不是挺管用

目前新的comment https://github.com/29988122/Fate-Grand-Order_Lua/commit/d7a58bc4772150553487c2f01e505cfaf26090f7 改了weak和resist的圖，Buster/Arts/Quick還沒動。您先測測看看行為與想像的吻不吻合。

== 如果不行，我們可能要另尋辨識良方，會花不少時間。 問題在於中文字筆畫多，1280*720只是單純pixel by pixel比對相似度的錯誤可能性太大了。

或許我去看看teserract/open cv之類的是怎麽實作灰階的圖像辨識吧...

[TW-Raino]

我覺得這個問題應該沒有這麼複雜，BQA卡的顏色剛好就是RGB。 直接抓指定位置的像素來檢查RGB的值就可以判斷是哪種卡了。

Local loc = Location(x, y)

Local r,g,b = getColor(loc)

if(r > g and r > b) then return B; else if(g > b) then return Q; else return A; end

大概這樣?

Jerry Ho [email protected] 於 2018年9月30日 週日 下午9:39寫道：

我不確定能不能行，因為Ankulua的天限，還有這字實在太小了，辨識不是挺管用

目前新的comment d7a58bc https://github.com/29988122/Fate-Grand-Order_Lua/commit/d7a58bc4772150553487c2f01e505cfaf26090f7 改了weak和resist的圖，Buster/Arts/Quick還沒動。您先測測看看行為與想像的吻不吻合。

== 如果不行，我們可能要另尋辨識良方，會花不少時間。 問題在於中文字筆畫多，1280*720只是單純pixel by pixel比對相似度的錯誤可能性太大了。

或許我去看看teserract/open cv之類的是怎麽實作灰階的圖像辨識吧...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/29988122/Fate-Grand-Order_Lua/issues/75#issuecomment-425721705, or mute the thread https://github.com/notifications/unsubscribe-auth/Apf6VeU5WAusst16cDWsRF8jBsCrHl55ks5ugMmogaJpZM4XAm6a .

[29988122]

其實我一開始有想過這樣做，但是他那個weak/resist的圖案會上下飄，同時卡片字樣也會跟著飄

雖然就算有飄，跟您說的一樣，如果用rgb來判斷的話，感覺多試幾次多觀察一下，應該也是可以找到一點，該點在漂移最低點<=Y pos<=漂移最高點，的區間都只有卡片顏色的紅藍綠，而不是字樣或是servant頭像的

不過這樣還是沒法解決weak/resist/克制/抵抗的伺服器語系字樣判斷問題..所以我才在想應該有更好更快的解法，想說找到再來一次處理，但是目前不知道該從何下手QQ

~ 不知道您有沒有頭緒？畢竟我也沒有CN帳號不好測，如果您願意送個PR來改善圖形辨識的準確度，刷箱中的master們應該都會感謝您的 XD

[TW-Raino]

weak/resist 有很明顯的圖形 [image: image.png] [image: image.png] 儘管圖形會上下飄動，某些座標點還是可以抓到正確的顏色。

Local loc_w = Location(x_w, y_w) Local loc_r = Location(x_r, y_r)

Local r_w,g_w,b_w = getColor(loc_w) Local r_r,g_r,b_r = getColor(loc_w)

Local weak = false Local resist = false

if(r_w > g_w and r_w > b_w and r_w > 160) then weak = true end

if(b_r > a_r and b_r > g_r and b_r > 160) then resist = true end

if(weak == true and resist = false) then return weak else if(weak == false and resist = true) then return resist else return normal

大概是這樣?

就算要圖形辨識，正三角形與倒三角形的邊緣比起文字更加清晰明確，應該優於辨識文字的影像。

Jerry Ho [email protected] 於 2018年10月1日 週一 下午4:49寫道：

其實我一開始有想過這樣做，但是他那個weak/resist的圖案會上下飄，同時卡片字樣也會跟著飄

雖然就算有飄，跟您說的一樣，如果用rgb來判斷的話，感覺多試幾次多觀察一下，應該也是可以找到一點，該點在漂移最低點<=Y pos<=漂移最高點，的區間都只有卡片顏色的紅藍綠，而不是字樣或是servant頭像的

不過這樣還是沒法解決weak/resist/克制/抵抗的伺服器語系字樣判斷問題..所以我才在想應該有更好更快的解法，想說找到再來一次處理，但是目前不知道該從何下手QQ

~ 不知道您有沒有頭緒？畢竟我也沒有CN帳號不好測，如果您願意送個PR來改善圖形辨識的準確度，刷箱中的master們應該都會感謝您的 XD

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/29988122/Fate-Grand-Order_Lua/issues/75#issuecomment-425833191, or mute the thread https://github.com/notifications/unsubscribe-auth/Apf6VZjz5jSlU_wRGyq_7hXeeXJMDphqks5ugdcFgaJpZM4XAm6a .

[potchy]

I have no idea what you guys are talking about, but see #46 and #50. Maybe it's the same problem.

[ryuga93]

Basically CN server has a preference of changing the font regularly, making region recognition harder. One solution is to change it to point recognition.

[PokeGuys]

我不確定能不能行，因為Ankulua的天限，還有這字實在太小了，辨識不是挺管用

目前新的comment d7a58bc 改了weak和resist的圖，Buster/Arts/Quick還沒動。您先測測看看行為與想像的吻不吻合。

== 如果不行，我們可能要另尋辨識良方，會花不少時間。 問題在於中文字筆畫多，1280*720只是單純pixel by pixel比對相似度的錯誤可能性太大了。

或許我去看看teserract/open cv之類的是怎麽實作灰階的圖像辨識吧...

OpenCV Object Extraction

Ankulua is already using OpenCV matchTemplate (You can decompile their APK and find the Imgproc.java). The only difference is maybe it can solve the scaling problem through python script. Or using the edge detection to identify the object. Compared to classic matchTemplate, it seems better and more accurate.

Method Comparison

The demo is running under python script, but using the same method as Ankulua

Tesseract 4

And also, due to the resolution of FGO, it is too small and too many noises next to word for Tesseract. It required a lots of preprocessing work before you can truly apply any OCR on it. (But I am using it to OCR the quest name and return the best matched info through the API tho)

Summary

So, I agree that recognizing the triangle is way more better than OCR.

btw, I am considering to build an API to process all the data and pass it back to the script to enhance the performance. It only takes about 50-100ms which is about 50-70x faster than the smartphone.

[potchy]

@TW-Raino Please try the latest commit to see if it solves the issue.

@pokeguys Frankly speaking, I have no experience whatsoever with the libraries you mentioned. 😅

While I do appreciate your ideas, I feel like this this project should be kept free from 3rd party APIs and network calls, so people with slow internet connections (myself included) don't experience subpar performance.

If you do come up with something handy, please open an issue/pull request or message us so we can take a look at it. As seen here, there are some functions in AnkuLua for making HTTP GET and POST calls. I'm not sure if it possible to upload screenshots this way, but I hope it helps you somehow.

[PokeGuys]

@potchy I think the API should deploy by the user and it should be connected locally (Nox <-> Ankulua <-> API). 🤔 And yes, it can pass the file through HTTP GET / POST request. I have already write this part.

Anyway, I will open another issue and explain it in detail.

[TW-Raino]

Hi, All

1. CN was update GUI at 10/24 so some image couldn't recognized now. I update some image to fixed it.

2. I also try to update the image of card type image(art / buster / quick ) but Ankulua could not distinguish between A card and B card.

[potchy]

1. Thanks, I already replied you in #84.
2. I updated weak.png and resist.png 3 days ago. I'm still waiting for confirmation whether issue was solved or not.

[Xianbei233]

so far so good, but maybe the card choice has some issue(not actually follow the BAQ, but is always what i want to choose

[potchy]

Unfortunately, I don't play the CN server, so I can't say why it's not working. It would be nice if you could record a video showing the problem or provide a transfer code for a dummy account so I can do a bit of testing for myself.

[Xianbei233]

@potchy https://drive.google.com/file/d/1wiLL79JECmHxlElyWCRDz9iMbjKRkJPo/view?usp=sharing

[potchy]

I finally managed to create an account in the CN server. Try my latest commit to see if it's finally working for everyone.

https://streamable.com/y1red