When converting a large batch of songs, some will fail to convert at first, but succeed after trying 2/3/4 times

[vivaria]

In https://github.com/vivaria/tja2fumen/issues/37, I tried converting every song in ESE (~2200 songs).

I found that many songs (~50 or so) would fail on the first pass, but then some of those songs would succeed when booting TDMX a second, third, or fourth time.

I'm not sure if the song itself matters, or if there's some sort of non-deterministic behavior stemming from the fact that TakoTako uses parallel processing for TJA conversion:

https://github.com/Fluto/TakoTako/blob/c269bcab60530877a16c2a473c84796b94d0a5ce/TakoTako/Patches/CustomMusicLoaderPatch.cs#L180

[Fluto]

That's a lot of songs 😅. Ideally, it should not matter if the software is run in parallel or not, but perhaps this issue is CPU dependent? Or perhaps there is some weird hardware/file locking that causes this non-deterministic behavior. I suspect some sort of file locking would be the main issue for this, or windows silently not allowing so many processes to run.

Some options would be to

• Investigate and fix the core issue
• Convert the files in sequential order (but that's giving up!)
• Convert files in the background and refresh the UI in game when loading a new scene (or something)

[vivaria]

Investigate and fix the core issue

I think I've found the core issue. I think it has to do with the new "tja2fumen migration" commit (https://github.com/Fluto/TakoTako/commit/f728d694b38da732d06a687b14ebbe676a6ff435):

I was debugging a separate issue (https://github.com/vivaria/tja2fumen/issues/39), and I found that TakoTako will try to convert songs using tja2bin.exe even when tja2fumen.exe is present.

(In that issue, I noticed some weird BPMCHANGE behavior that occurs with tja2bin but not tja2fumen. So, I deleted the converted song then reconverted, and the song was fixed, confirming that tja2bin was used by mistake during the first batch of conversions.)

I'm still trying to figure out what the exact circumstances are where tja2bin.exe will be used? My hunch is that is has something to do with the parallel execution. For example, if tja2fumen.exe is currently in use, then maybe the Directory.enumerateFiles method fails to find it?

From the docs for Directory.enumerateFiles, we see the following under Exceptions:

UnauthorizedAccessException The caller does not have the required permission.

So, I'm wondering if maybe this exception gets thrown, and thus GetTja2FumenPath continues on to check whether tja2bin.exe exists.

And, maybe tja2bin.exe is always found, because a simple File.Exists check is being used on the hardcoded path instead.

---

Anyway, if we assume that TakoTako is falling back to tja2bin.exe for 3 songs in each batch of 4, then maybe this issue (#26) is because:

• When I first process the big batch of 2200 songs, 3/4 of the songs are actually being converted by tja2bin.exe and not tja2fumen.exe.
• Hence some failures, because tja2bin.exe has several bugs of its own
• But, when I re-run the batch conversion, the amount of songs being converted changes, and then perhaps tja2fumen.exe is used for those failed songs on the 2nd/3rd/4th passes.

[vivaria]

I found that TakoTako will try to convert songs using tja2bin.exe even when tja2fumen.exe is present

Yes, this is definitely the cause.

• I removed all 2200+ conversions that I did.
• I then made a copy of tja2fumen.exe, then replaced tja2bin.exe with it. That way, if TakoTako does try to fall back to tja2bin.exe, it will actually be using the renamed tja2fumen instead.
• I then converted all 2200+ songs again.
• This time, retrying the conversion had no effect on the # of failures. The same songs failed every time. So, multiple passes were not needed.

Now to figure out why TakoTako is falling back to tja2bin.exe in the first place when it shouldn't be. 🤔