Support custom fonts for TextMeshPro

[gravydevsupreme]

Many games uses TextMeshPro with a font that has a very limited character set. When translating text into another language the translated text likely cannot be shown due to this.

[gravydevsupreme]

Examples Issues:

• #53
• #35

[Cerzs]

Hi @gravydevsupreme just curious, any ETA for this feature? I've found some games with TMP, mostly from MangaGamer.

Ty!

[gravydevsupreme]

It's not really an issue I have looked at it.

The problem with it is that it requires a bit of research on my end on how the tooling behind TMP works.

And since I don't really work with Unity or even it's Editor at all in my day-to-day business, it's not really something I have gotten a chance to look at yet.

Certainly I would like to look at it. but I cannot currently provide an ETA, especially since there are a lot of unknown that I am very uncertain of how much effort takes:

• How to generate the font using their (probably) native library
• How to actually get unity to load it, since these things are usually instantiated in the native part of the engine. (this thread suggest it is possible to load them through asset bundles, which may be the best option http://digitalnativestudios.com/forum/index.php?topic=658.0).

Other potentially relevant reference: http://digitalnativestudios.com/forum/index.php?PHPSESSID=kc1ilgtph5b096bd4j9v9v5fg4&topic=805.0

[gravydevsupreme]

Just a small update on this one:

Today I tried to build the fonts that comes when you install the standard TextMeshPro package into an asset bundle and load them through the plugin. And it did seem to work. At least with that specific version of the engine.

So there may be some progress towards this goal.

But that means I would need to figure out which unicode ranges I would need to include in such a font to support various languages.

Unicode table: https://jrgraphix.net/research/unicode_blocks.php

Essentially I would want ranges for: Japanese, Chinese, Korean, Russian + various uncommon latin characters. (Maybe I am forgetting some).

Frankly, I could use some help with this, because some of these unicode ranges seem huge. For instance is all Hangul characters in the range AC00 - D7AF really used?

And I guess I would have to do some compromising when it comes to the thousands of Kanji characters that exist.

I tried generating a font that included thousands of characters today, but the progress bar never left 0. ^^

[Cerzs]

Hm, maybe doing it in parts? Spanish uses these characters only, for example.

Including various languages into a font maybe would create a very large font, 10+ mb perhaps?

[gravydevsupreme]

So, I got a prototype working after spending way too much time looking for fonts.

I have created two fonts that can be placed in the root game directory and referenced from the config file through the TextMeshPro config option.

• arialuni_sdf (I think in general this supports most characters)
• notosanscjk-regular_sdf (may not support russian)

Place the unzipped font files in the game root and set config OverrideFontTextMeshPro=notosanscjk-regular_sdf

There is a significant chance that this is not going to work in all games and it probably depends on both the version of the Unity Engine that is used and the version of TextMeshPro.

These files are generated using Unity 2018.1 and the TMP that comes with it and I can confirm it works with that ResizeMe game from #69

fonts.zip on mediafire XUnity.AutoTranslator-ReiPatcher-4.7.1.zip XUnity.AutoTranslator-BepIn-5x-4.7.1.zip

These asset bundles can be generated without programming knowledge, but the older the version of unity to target, the more difficult it may be.

I would like to know, if you can tell me, where it works and where it doesn't.

You can see which version of unity is used by right clicking on the game exe file and going to Details. From there, there is a File version and Production version which represents the unity engine version.

[Cerzs]

I've tested with Cartagra, the game just keep freezed. I'll try with other games. Rei does the same thing.

Unity version:

Also getting errors but running anyway.

System.TypeLoadException: Could not load type 'BepInEx.Preloader.RuntimeFixes.UnityPatches' from assembly 'BepInEx.Preloader, Version=1.0.0.0, Culture=neutral, PublicKeyToken=null'. at BepInEx.Preloader.Preloader.Run () [0x00000] in <filename unknown>:0

Tested games (Freezed):

• Cartagra
• Free Friends

[gravydevsupreme]

Here's a new version that fixes the freezes (the plugin was using a "new" API not available in these old games):

XUnity.AutoTranslator-ReiPatcher-4.7.1.zip

I can imagine BepInEx not working particularly well on such old games.

Now, just because it doesn't freeze doesn't mean it works.

The problem is just substituted with another, and that is the fact that this asset bundle is not compatible with the game because it was generated by a much newer version of the Unity Editor.

The following message appears in output_log.txt:

This asset bundle was not created with UncompressedAssetBundle flag, expected id 'UnityRaw', got 'UnityFS'

Which likely means that if we can actually generated a bundle that is compatible with this engine, it is going to work.

Such a process would involve:

1. Obtaining an appropriate Unity Editor from https://unity3d.com/de/get-unity/download/archive
2. Creating a new project
3. Installing a TextMeshPro version that is compatible with that Unity Editor
4. Creating a TextMeshPro font asset from a "standard font" (good choice is "Arial Unicode MS Regular" or "arialuni" which I think can always be found in the windows Fonts folder.
5. Creating an asset bundle that contains the created TextMeshPro font asset.

I am not sure how compatibilities look across different versions of Unity when it comes to asset bundles. Maybe an asset bundle generated in 5.x or 4.x will be compatible with 2017 or newer. Or maybe not. Same thing can be said of the actual generated TextMeshPro font asset.

[gravydevsupreme]

This might sound a bit easier than it is though. Because TMP was only included as part of Unity since version 2018. Prior to that it was a separate paid product.

[gravydevsupreme]

Actually, I wanted to try to generate an asset bundle that isn't compressed and see if that works before attempting anything else. But right now my Unity Editor is acting up and crashing whenever I open it...

[Cerzs]

Yep, the game is now working, but as you say, is not compatible with this old version.

[gravydevsupreme]

Tried with an uncompressed bundle. Doesn't work either.

[gravydevsupreme]

Here's a new fonts upload that includes a new font asset bundle that should theoretically work for unity 5.5 - 2017.4. Theoretically as in not really tested.

Still wont work for Cartagra. ~~Frankly, I cannot even get the Unity 4.x Editor to run on my system.~~

https://www.mediafire.com/file/03f3ru5d8hk3rq9/fonts.zip/file

Got Unity 4.5 working on my system. But it turns out that building asset bundles requires the professional edition. -_-

[Cerzs]

I'm trying to find a game 5.5 or newer, but all that I found is 4.x-5.3

Free Friends 2, for example:

[djlaser]

Initialize engine version: 2018.3.4f1 (1d952368ca3a) plugin worker (Russian translation) from arialuni_sdf

[gravydevsupreme]

Getting this to work for old games (pre-5.5) is pretty much a no-starter unless there is some way to obtain an older version of TMP.

There's several problems here:

• The asset store offers zero revision history on uploaded assets. (I don't even know what to say about this. So absurd)
• The previous way to obtain TMP, which was through the creator's private forum has been disabled. You would previously purchase TMP through that forum. The author did state that if one had previously purchased the asset, you would still be able to access it there.
• I have absolutely no clue how I would obtain these old versions of TMP in any other way than actually contacting the author, which I don't really feel like doing.

And if by some miracle we manage to obtain an old version of TMP, we would also need a professional edition of the old Unity Editor to support pre-5.0.

[gravydevsupreme]

The latest version (4.8.0) now has support for loading external asset bundles.

With that being said, I had trouble getting these bundles to work with anything that weren't based on Unity 2018 or greater where the Unity.TextMeshPro.dll file is present in the managed directory.

The asset bundle loading implementation is probably going to work for any game, but the problem is still generating the correct asset bundle with a compatible asset. I fear that for older version of Unity it might require exact version matches for TextMesh Pro and (potentially) Unity, which is a massive pain.

[gravydevsupreme]

Interesting to see you could get it working in a Unity 2017 game.

When I generate a font I usually use the following settings:

• Source Font File: Arial Unicode MS Regular (arialuni.ttf). Noto Sans CJK might be a better choice for you, although I think arialuni does have most of the important Kanji.
• Sampling Point Size: 40
• Padding: 3
• Packing Method: Fast
• Atlas Resolution: 8192 x 8192 (yes, that's a big texture, but I am trying to include many languages as well)
• Character set: Uncode Range (Hex) 0000-052F,2E80-2EFF,3000-4DBF,4E00-9FFF,AC00-D7AF
• Render Mode: SDF16 / Distance Field 16

Usually takes round about half an hour to generate.

[gravydevsupreme]

When it comes to the GUIDs of TMP, I think they can be separated into:

• 2017 and earlier
• 2018 and later

This blog post describes how these GUIDs were modified during that transition.

Although if that was the only issue I would kind of have expected the Unity 2017 asset store TMP / and the "loose" TMP to be compatible, which they clearly are not. This is really my main concern because such arbitrary differences are going to be difficult to handle, if it's supposed to be done in an automated fashion.

After simply generating a TMP asset bundle for 2018.1, I was kinda surprised to simply see it working in a 2018.4 game, since I assume the package manager TMP should suffer from the same kind of problems (minor differences in data format expectations).

Generally I would say that the best approach is to simply target the specific version of Unity / TMP that a game uses.

It's very cool to see you got it working with Emotion Creators.

[djlaser]

5.6.2f1 and 5.6.4f1 Dont work TMP output_log.txt output_log2.txt

[gravydevsupreme]

There isn't really that much I can do about this.

I do not expect the fonts I have included in the release section supports any version of TMP where there is not a dll called Unity.TextMeshPro.dll in the Managed folder.

The problem is that I am unable to build the font without access to the editor tools for the version of TMP used in that game. And those old versions of TMP are not free and not even available for purchase anymore.

[djlaser]

There isn't really that much I can do about this.

I do not expect the fonts I have included in the release section supports any version of TMP where there is not a dll called Unity.TextMeshPro.dll in the Managed folder.

The problem is that I am unable to build the font without access to the editor tools for the version of TMP used in that game. And those old versions of TMP are not free and not even available for purchase anymore.

Thanks for the answer. The app is very good but why the there is no in Diskord canal on him.

[gravydevsupreme]

This plugin is developed primarily in relation to various ILLUSION games. As such it is primarily discussed on Illusion Soft and Koikatsu! discords. I'm scrubbing around on those discords regularly.

There is no discord for this plugin specifically, as I don't really want to maintain it nor do I really feel like it is warranted.

[djlaser]

Hello dear gravydevsupreme!

Please write instructions or make video  how to generate TextMeshPro fonts and it is desirable to how to collect the generated script assestsbundle

Sincerely, djlaser That's what I did and then did Fonts1.zip

[Cerzs]

I was wondering, some games includes .ttf fonts in their .assets, is not possible to redirect the TMP to that .ttf file?

[gravydevsupreme]

I'm afraid not. TMP requires assets that are build specifically for it. You can construct such assets from .ttf files using its editor tools. But as I've mentioned before you generally need the correct version of the tooling to be able to do that. Tooling, which for old versions, is not available for purchase anymore.

[Cerzs]

This may not be useful, but TMP textures can be read/saved with a Photoshop NVIDIA plug-in.

The bad news: You have to modify the symbols you don't need for the character you want. Example: * = á.

Also, this may break the font slightly.

[gravydevsupreme]

It also looks like that texture is built based on the actual characters used in the game (as opposed to unicode ranges). So replacing any of them would cause that character to be missing where it was previously used.

I'm not sure I am ready to jump down the rabbit hole of manually manipulating textures like this.

If I was extremely desperate to get this game translated, then my approach would probably be to contact the author of TMP (which I am pretty sure is just a single person (at least back then)) and ask him how much it would cost to get access to the archived versions of the TMP editor/runtime tooling. (of course, with the risk of still not being able to get it working after paying)

This is of course still assuming that the Unity version is greater than 5.x. If not you need access to a professional edition of the Unity editor as well.

[nathan60107]

Hi, I just want you to know the solution at https://github.com/bbepis/XUnity.AutoTranslator/issues/57#issuecomment-562344581 work on the game Skul. Its file version is 2019.4.8.30749, greater than 2018. So it works correctly as expected.

The solution helps me a lot. I think you should put it at release and TextMeshPro=arialuni_sdf should be a default config. Or at least, put a link at README.md to let the user know if their game use TextMeshPro and their game cannot display translation text correctly, they can try this solution to solve it. This issue always happens in the area which the native language is not English.

Thank you for create XUnity.AutoTranslator.

[gravydevsupreme]

It certainly isn't the most visible part of the documentation, but it is documented.

Although it can certainly be daunting to find anything in that README.