Fate-Grand-Automata
Np level detection
Hello, I thought of this feature because I'm going to farm Gilfest and I need a friend np5 jarcher. I reckon adding a np level check should be doable? It seems like a very beneficial addition. Thank you Dante
Seems like this isn't as simple as I thought initially now that I looked at many support servants.
These are the challenges:
- Length of NP name causes the NP level number to be at different positions.
- The NP level can be in 3 different colors depending on Not-friend/friend/Interlude or Rankup done.
- I don't recall if NPs have numbers in their name which could be wrongly detected as NP levels.
- The NP name shifts depending on whether master message is set. This one is simple to handle, just added here so I don't forget.
One servant I can think of that has numbers on their NP is summer Kiyo (Dojoji Bell, Method 108 - Fire Dragon Mower), seems like the detection would have to include the "Lv. "
@tchofy Ah, I'm dumb, nice catch, Lv is present for everyone.
So, only checking Lv. 5 in both white and yellow should be good enough (at-least for NA).
When NP 5 is asked for, we should assume Friends only.
@reconman In that case, this won't work without text recognition 😢.
Hmm these seem like tough challenges. Perhaps as an alternative, you could only check for np5(and friends obviously, otherwise you can't use np)? So you just check if there's yellow for like the first letter. I don't know if it's possible, but just suggesting the idea.
@Dante0909 Missed your last comment. The colour isn't the problem here, it's the varying font size like @reconman pointed out which we don't have a way around right now.
This type of problem is the reason I initially implemented the Friend Name feature. Since it is annoying and difficult to deal with all of these problems, I went with a simple route where you can just use friend names if they are the ones with the NP levels you need. If it is possible for you to be in contact with them personally, you can ask them to keep that servant up. Alternatively, the contributors can always mix the Friend Name feature with the Servant/CE preferred feature and allow a cross examination between the Friend Names you selected and if they have the Servant/CE you are looking for that you know they have.
This shouldn't be too difficult to implement.
@LouizFC with OCR, the location of "Lv. X" would need to be specified, which varies depending on how long the NP's name is. If the rest of the text is not cropped out correctly, OCR would give bad results, more so for Chinese, Japanese, Korean texts if we're using an English OCR. I had tried using OCR for Skill level checks but the results weren't good, so I resorted to Image matching there too. But, image matching won't work here since the font size varies depending on the length of NP name.
@MathewSachin I forgot about Chinese, Japanese and Korean texts, sorry. I have been playing with some lightweight neural network image recognition at work, when I get some time I will run some tests. Do you know how I can get my hands on some ingame text samples from the mentioned languages? (aside from creating a new account on each region, of course. But if that is the only option I will go with that too)
Edit: Maybe we could get the whole NP name as an image, but train a very basic model to recognize only the Lv. X text, but I need to know how well it would perform on a regular mobile phone, my phone for example is pretty old
I experimented with some custom Tesseract training for a different subject today and got very good results. When I get some time I pretend to:
- Collect some FGO samples from videos.
- Test on a simple PC implementation
- Port the implementation to an android friendly module.
@reconman Exactly. Sorry about that. In my native language there is a word in verb form called "pretendo", which translates to "intend to" in english, I though it was a cognate, but looking up now I was wrong. Thanks for the correction
Just saw you mentioned Chinese. I have a TW account, can I help in any way?
Just saw you mentioned Chinese. I have a TW account, can I help in any way?
@the3dsandwich thank you for wanting to help. I need some screenshots from your friends support list of some specific servant np names:
- Kyohime summer - because it has numbers in the name
- Anne & Mary Summer - same reason as Kyohime
- Assassin Shiki - very long name that distorts the np "Lv" text.
Are you matching the English part of the names? Chinese numbers are drastically different from normal numbers that I don't think it'll affect image matching. Also I'll be on the lookout for long NP names (assassin Shiki's NP name is on the short side)
Wait. Lemme upload some screenshots. Screenshot file big, I need shrink.
I know you asked for the lancer version but none of my friends have a lancer kiyo atm and regular kiyo also has a number in her NP.
Longest NP name I can find atm
Stella in TW server also have a number 1 in it.
As you may see now TW NP names are generally short, but I'll be on the lookout for long NP names. Also I think Chinese numbers is different enough to not cause an image recognition problem?
If TW NP names are always short, simple image recognition should be enough. If only other servers could be the same case 😢 .
Also I think Chinese numbers is different enough to not cause an image recognition problem?
Image matching would indeed fail by just minor font changes, but, since LouizFC is trying OCR, I think number font changes might not be a problem.
Edit: It seems Chinese numbers are completely different, so that eliminates the problem I guess.
@the3dsandwich thank you very much for the samples. I will try to manually "resize" the "Lv" text just in case.
@MathewSachin before I work on this, is there any objection in using Tesseract in the app? because if yes I can try other lightweight approaches first
I'm fine with Tesseract if it gets the job done. I believe the APK size increase won't be as drastic as when adding x64 libraries (#624).
I had tried Google on device ML APIs for detecting skill levels before I resorted to image matching because the results weren't satisfactory. Well, I'm just not good at ML stuff in general.
@MathewSachin I am also not good at ML stuff, I began studying it in the past month. I got interest in this issue as a learning oportunity
It feels like there should be good solution for OCR-ing numbers (from a known font) on-device, but even ignoring that, can't we at least just get NP5 detection? This would already cover a lot of cases (when farming relies on maxed support DPS or 4*) and you only need to detect that font is yellow.
Btw, did a quick and dirty test looking for Lv. ?(\d) and tesseract detectText was reasonably okay although it sometimes would confuse 1 and T, but googles mlkit appeared to be way more accurate. Perhaps it's time reevaluate the feasibility.