New translation format system

[HenryLoenwind]

This is a replacement for #1134 and supersedes #1269.

#1134 was a good idea, but it ran into the issue that its translations are not editable on Crowdin and will even be trashed unless a translator knows exactly what they are dealing with.

This new system doesn't employ json structures but operates on the string contents of a translation. If it starts with a special marker (a sequence that is invalid for vanilla code), the translation string will be parsed by our parser instead.

The format is inspired by html and MiniMessage. It supports:

• Markup for all ChatFormatting styles, e.g. This is <red>red</red>!
• Aliases for ChatFormatting: b, i, em, u, o, obf, s, st, grey, dark_grey.
• Parameter insertion, e.g. <0> <1> <2>
• ICU plural support, e.g. <0:plural:one{is # dog} other{are # dogs}>
• Inclusion of other language keys, e.g. <ref:other.key>
• Keybindings, e.g. <key:key.jump>
• RGB colours, e.g. <color:4637>...</color> <color:#FF00FF>...</color>
• Fonts, e.g. <font:minecraft:alt>...</font>

It is intended that this replaces #1134 in the next breaking phase.

Custom placeholder for Crowdin: start, then "<", anythingbut ">", then ">", end (untested)

---

Note: This could also be combined with this, i.e. registerable format handlers. ("%n(neoforge:html) <red>boo</red>", "%n(neoforge:sjson)[{\"text\":\"boo\",\"color\":\"red\"}]", "%n(mymod:bbcode) [red]boo[reset]", "%n(othermod:markdown) **boo**", ...)

[neoforged-pr-publishing[bot]]

• [x] Publish PR to GitHub Packages

Last commit published: bc117bd512a25bbdd022a678ca2bbde13933ba1e.

PR Publishing

The artifacts published by this PR:

• :package: net.neoforged:neoforge:21.0.161-pr-1408-i18next
• :package: net.neoforged:testframework:21.0.161-pr-1408-i18next

Repository Declaration

In order to use the artifacts published by the PR, add the following repository to your buildscript:

repositories {
    maven {
        name 'Maven for PR #1408' // https://github.com/neoforged/NeoForge/pull/1408
        url 'https://prmaven.neoforged.net/NeoForge/pr1408'
        content {
            includeModule('net.neoforged', 'neoforge')
            includeModule('net.neoforged', 'testframework')
        }
    }
}

MDK installation

In order to setup a MDK using the latest PR version, run the following commands in a terminal.
The script works on both *nix and Windows as long as you have the JDK bin folder on the path.
The script will clone the MDK in a folder named NeoForge-pr1408.
On Powershell you will need to remove the -L flag from the curl invocation.

mkdir NeoForge-pr1408
cd NeoForge-pr1408
curl -L https://prmaven.neoforged.net/NeoForge/pr1408/net/neoforged/neoforge/21.0.161-pr-1408-i18next/mdk-pr1408.zip -o mdk.zip
jar xf mdk.zip
rm mdk.zip || del mdk.zip

To test a production environment, you can download the installer from here.

[HenryLoenwind]

Here is the monster I've been using to test:

"neoforge.configuration.title": "%n <blue>%s</blue><color:#7700ff00>gr<underline>e<key:key.jump>e</underline>n</color> <font:minecraft:alt>f<u>o</u>nt?</font> (There <1:plural:one{is # dog} other{are <blue>#</blue>\\> <italic>dogs</italic>}> here.) <ref:fml.menu.mods.config>",

It generates:

open TODOs/questions:

• [x] %n -> %f. --- No. There's too much magic going on here, %f gets transformed to %s at a low level. Not worth messing with that when %n works fine.
• [x] I18N/other places that may access the raw language value?
• [x] filter in net.neoforged.fml.i18n.FMLTranslations?
• [x] an automated test
• [x] convert all json-format keys in neo's langfile (I volunteer)
• [x] docs
• [ ] other formatters? numbers, dates, spellout, ordinal, duration?
• [ ] report exceptions to the logfile? (vanilla doesn't)
• [x] other Style attributes? (click, hover, insert) -- No. Those only make sense for chat.

[Technici4n]

Thoughts on just having the stringized version of the JSON in a string (with some prefix) instead of this complicated alternative syntax? (Similar to SNBT for recipes before data components).

[HenryLoenwind]

Thoughts on just having the stringized version of the JSON in a string (with some prefix) instead of this complicated alternative syntax? (Similar to SNBT for recipes before data components).

It wouldn't be less complicated as we cannot simply throw that json back into the parser and have it end up in the language. We'd need to iterate and execute it to produce a list of FormattedText text anyway. And at that point, there's no real advantage anymore. On the other hand, JSON is a pain to edit, whereas this format is much more straightforward. "<red>Hello <0></red>" vs "[{\"text\":\"Hello\",\"color\":\"red\"},{\"index\":0,\"color\":\"red\"}]"

[lukebemish]

Correct me if I'm wrong, but the parameters substitution approach Techni4n mentions requires stuff on the code end, does it not? Thus, it is not viable for translations where the key and how it is consumed are all generated based on some RL on the vanilla end -- which is a majority of translations! Furthermore, this system seems to support stuff like the plural setup that is much more complicated, if not impossible, to handle on that end as how it works varies by language. You simply cannot define it via a substitution like that -- you need language-specific logic. As such, what Techni4n proposed as option 1 is just not a viable option for solving the issues this PR solves -- whether or not this PR is the right approach to solving those issues.

[Technici4n]

I find your assessment that option 1 is not viable... interesting... considering that Minecraft is using it just fine.

[lukebemish]

The scope of what mods want to do and need to do is far larger than the scope of what vanilla does or needs to do. Your assessment that vanilla is using it just fine is irrelevant when I have pointed out situations it does not work in in the much broader modded context. There's a reason, after all, that the JSON embedded components in translation files were implemented to begin with.

[TelepathicGrunt]

@lukebemish if we are going to drag in mods as an excuse, then we should be asking for real in-use examples of actual mods that need a different system than what vanilla does. If vanilla can handle the other languages without much issues in translating, then it seems unlikely that a modder would need to deviate heavily. Heck, there might not be any modders whose mod will live and die by special casing rules translations.

We can speak in hypotheticals all day and nothing gets merged or changed or cleaned up. Hence why if it is going to be argued that a system must be kept/used by modders, let’s get some actual examples of people who need a different system than vanilla’s

[lukebemish]

This isn't hypothetical. For a very real use of something like the plural system, see the fluid unit PR which ran into pain here. That sort of formatting varies by language, so doing it by in-code substitution results in writing a lot of different substitution cases, ends up with at least 3 keys per thing being translated, and doesn't play nice with RL-based translation keys like pop up there.

Furthermore: say I want to use this sort of formatting in the translation key for, say, a tag. Using substitution to do that is just quite literally impossible, as I don't control the code where the translation is used!

[HenryLoenwind]

ends up with at least 3 keys per thing being translated, and doesn't play nice with RL-based translation keys like pop up there.

To support plurals for all languages, you need 6 translation keys.

Vanilla doesn't have any translation string for normal gameplay that has a number in a sentence, they worded everything to avoid that. They have some in command output, where they use single values with "item(s)", which is acceptable in English (and Germanic languages in general) but gets really awkward in languages with more complex/more forms.

[neoforged-automation[bot]]

@HenryLoenwind, this pull request has conflicts, please resolve them for this PR to move forward.

[Matyrobbrt]

Given the lack of activity and the need for a rebase, I'll be closing this PR for now. Feel free to open it again after the conflicts are resolved.