Client Side translation of Server Messages

[edg-l]

The server would send a packet instead of a chat message with a id and arguments, then the client would display the translated messages with those values.

We would then need our own printf parser according to @heinrich5991

We also need to take care of compatibility with unsuported clients.

Any more ideas?

This is a step towards https://github.com/ddnet/ddnet/issues/269 too

[heinrich5991]

[7:35 PM] Learath2: Oh, btw how are we supposed to deal with unsupported clients? The version number is an ugly fix […] [7:35 PM] heinrich5991: I'd send the untranslated string as well [7:35 PM] heinrich5991: that'd be some network traffic though […] [7:36 PM] Learath2: @heinrich5991 and it would require very ugly string matching to remove the untranslated string [7:36 PM] heinrich5991: hm? no [7:36 PM] heinrich5991: I was thinking: [7:36 PM] Learath2: especially since there is nowhere to hide a mark in a chat message [7:37 PM] Ryozuki: maybe add a hidden unicode characte [7:37 PM] Ryozuki: at the start [7:37 PM] Ryozuki: and use it to hide on modern clients [7:37 PM] Ryozuki: superh acky hack [7:37 PM] Ryozuki: :nouis: [7:37 PM] heinrich5991: a new message type TRANSLATED_SERVER_MESSAGE, with an ID (UUID), untranslated message (string), format parameters (various, string, int) [7:37 PM] Ryozuki: but the problem here is old clients [7:37 PM] Ryozuki: they dont know about this packet [7:37 PM] Ryozuki: they only want chat messages [7:38 PM] heinrich5991: yea, distinguish them by version [7:38 PM] Ryozuki: then why we need to send the untrasnlated messages o nnew clients? [7:38 PM] Learath2: The new message type is simple [7:38 PM] Ryozuki: i dont find the reason [7:38 PM] Learath2: Ah by version [7:38 PM] heinrich5991: because they might not support all the strings we need [7:38 PM] Ryozuki: damn i type like a retard [7:38 PM] Ryozuki: missing keys [7:38 PM] Ryozuki: xd [7:39 PM] Ryozuki: ah thats true [7:39 PM] Ryozuki: but [7:39 PM] Ryozuki: it can fallback to the english [7:39 PM] Ryozuki: ah [7:39 PM] Ryozuki: we dont have a english [7:39 PM] Ryozuki: its directly in the src [7:39 PM] heinrich5991: yes, that's the untranslated string :wink: [7:39 PM] Ryozuki: well then im fine with that [7:39 PM] Ryozuki: btw add any final decicion to the issue [7:39 PM] Ryozuki: this will be lost [7:39 PM] Ryozuki: here [7:39 PM] heinrich5991: yes, true [7:39 PM] heinrich5991: thanks for the reminder [7:39 PM] Learath2: So we check the version, if it's too old we send preformatted english string. If new enough to know this string, send unformatted message with parameters [7:40 PM] Ryozuki: and english [7:40 PM] heinrich5991: no, if new enough to know the TRANSLATED_SERVER_MESSAGE message [7:40 PM] heinrich5991: then we send it [7:40 PM] Learath2: Ah, you want to send the entire string [7:40 PM] heinrich5991: then the client can decide whetheri t knows this particular message [7:40 PM] Ryozuki: in case its not translated [7:40 PM] Ryozuki: we need a fallback [7:41 PM] Learath2: meh, atleast it's not more traffic [7:41 PM] heinrich5991: well, my solution is more traffic?! [7:41 PM] Ryozuki: yes its more [7:41 PM] Ryozuki: but i dont think it matters [7:41 PM] heinrich5991: the UUID + the formatting parameters [7:42 PM] Ryozuki: why a UUID? [7:42 PM] Ryozuki: and not a simple id [7:42 PM] Learath2: It's sliightly more traffic [7:42 PM] heinrich5991: @Ryozuki because people are going to add their own messages and we don't want to conflict [7:42 PM] Ryozuki: ah [7:42 PM] heinrich5991: e.g. noby adds a message for fng [7:42 PM] Learath2: Why bother with an id at all actually? [7:42 PM] Ryozuki: you are thinking outside ddnet too [7:43 PM] Ryozuki: good i forgot [7:43 PM] heinrich5991: @Learath2 because the untranslated string might change without the translations needing change [7:43 PM] Learath2: We can send the untranslated string unformatted [7:43 PM] heinrich5991: (was my thought) [7:44 PM] Ryozuki: i prefer a uuid [7:44 PM] Ryozuki: this get stuff translation is rly funny [7:44 PM] Ryozuki: cuz there are repeated translations [7:44 PM] Ryozuki: atleast i found that when translating […] [7:45 PM] Learath2: I sort of dislike having redundant information, but I guess I can't come up with a better solution [7:45 PM] Ryozuki: if u cant then its not that redudant i guess [7:46 PM] heinrich5991: well it is redundant in the sense that we control client and server […] [7:46 PM] heinrich5991: but not redundant if we don't assume that [7:46 PM] Learath2: It's redundant because we actually know on the server whether a client knows a string or not via the version number […] [7:47 PM] heinrich5991: but I thinkn that's kind a ugly […] [7:48 PM] Ryozuki: btw remember to add info to the issue in github [7:48 PM] Ryozuki: xd [7:48 PM] Ryozuki: we will rly forget

[Banana090]

I'm really looking forward to that update 'cause I'm planning to use server messages for chat formatting, it's very useful. So we don't have to show server messages such as "Enter solo", "exit solo", "player XXX finished in YY:ZZ" in chat, but could do it any other way.

About this issue: Checking version on server side and then sending packet containing unformatted info with arguments for newest clients and simple string for older ones looks like a perfect solution for this.

[Learath2]

One problem with the approach discussed on discord is that we miss the opportunity to turn some things into actual events. Catching a TRANSLATED_SERVER_MESSAGE and translating it to an event would be a very ugly solution.

[heinrich5991]

@Learath2 Could we maybe rename it to SERVER_GAME_MESSAGE and treat the chat as a fallback? So we can always just display these messages somewhere else, e.g. in the kill messages.

[edg-l]

How would you approach sending the arguments?

Currently, our network data types don't allow such thing, specially considering if each argument can be str, int, float, etc.

	NetMessageEx("Sv_TranslatedMessage", "[email protected]", [
		NetString("m_Uuid"),
		NetString("m_PreformattedMessage"),
		NetString("m_Arguments"),		
	])

We could put all the arguments in a string separating them with some token, but we still won't know their type, also I don't like it.

We could also create a NetUuid datatype which could be useful maybe.

[Learath2]

You make the netmsg non size checked and just add the things to be formatted to the end.

On Fri, Nov 6, 2020, 17:57 Edgar [email protected] wrote:

How would you approach sending the arguments?

Currently, our network data types don't allow such thing, specially considering if each argument can be str, int, float, etc.

NetMessageEx("Sv_TranslatedMessage", "[email protected]", [ NetString("m_Uuid"), NetString("m_PreformattedMessage"), NetString("m_Arguments"), ])

We could put all the arguments in a string separating them with some token, but we still won't know their type, also I don't like it.

We could also create a NetUuid datatype which could be useful maybe.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/ddnet/ddnet/issues/3091#issuecomment-723187857, or unsubscribe https://github.com/notifications/unsubscribe-auth/AADXYBHIEXQBAEI2WPT4SOTSOQTITANCNFSM4SPKNVRA .

[TsFreddie]

Can we specify what language is the m_PreformattedMessage in instead of assuming it is English. Just in case some mod originate from a non-english community. So players can make English translations of those too.

[heinrich5991]

Sounds good, the m_PreformattedMessage is in any language, and there can be english translations, too.

[TsFreddie]

thought about this recently. Here's my take (again):

I still don't get how a UUID would benefit, and generating them across multiple mods/servers sounds like a huge pain. So, I still think unformatted might be the way to go (and multiple mods using the same format doesn't need to be translated again.)

Here's my idea for a protocol:

	NetMessageEx("Cl_SetLanguage", "[email protected]", [
		NetIntAny("m_Lang"),
	])

	NetMessageEx("Sv_TranslatedFormatString", "[email protected]", [
		NetIntAny("m_ID"), # simple id, stored client-side, used for referencing said format
		NetString("m_FormatMessage"), # unformatted in english
		NetString("m_Context"), # contextual info, mods can use this if they prefer not to use other mods translation.
		NetIntAny("m_Lang"), # can be -1
		NetString("m_TranslatedFormatMessage"), # can be empty
	])

	NetMessageEx("Sv_LocalizedChat", "[email protected]", [
		NetIntAny("m_ID"), # reference the format string
		NetString("m_Arguments"), # or whatever way of listing arguments that we ended up with.
	])

	NetMessageEx("Sv_LocalizedBroadcast", "[email protected]", [
		NetIntAny("m_IDLine1"), # reference the format string
		NetIntAny("m_IDLine2"), # reference the format string
		NetIntAny("m_IDLine3"), # reference the format string
		NetString("m_Arguments"), # all arguments concatenated
	])

Again, I strongly disagree not letting clients provide their language settings to servers.

The Cl_SetLanguage info can be used in three ways:

1. Identifying the localization feature so the server knows it supports localization without relying on ddnet version.
2. Allow server provided translation if there is one, based on users' language setting.
3. Enable translation for user(hoster) generated strings (rcon say, rcon broadcast, vote menu) to be translated. And on mods with abused vote menu (shop menu, upgrade menu, etc), which normally does not make sense to translate client-side.

---

To send a localized chat from server:

1. Make sure we identified client's language settings. If not, send Sv_Chat instead and skip all following steps.
2. Check whether the Message ID has already been sent to this user using their language * if yes, do nothing. * if no, send it via Sv_TranslatedFormatString (Populate m_Lang and m_TranslatedFormatMessage if server has translated version).
3. Send Sv_LocalizedChat with the Message ID and arguments.

---

To display a localized chat in client:

• Assuming we already sent at least one Cl_SetLanguage message upon join and language changes.

On Sv_TranslatedFormatString:

1. Find m_FormatMessage in our language file.
   • if found, discard m_Lang and m_TranslatedFormatMessage. (Meaning we already have a translation locally)
   • if not found AND m_Lang does not match our language setting, discard m_TranslatedFormatMessage.
     • if m_Lang is not -1, send Cl_SetLanguage to notify server about our language change again (because the previous one sent on language change has likely been lost).
   • if not found AND m_Lang matches, add m_TranslatedFormatMessage to our language, and mark the language file (which should be a separate file in storage TYPE_SAVE) dirty, so we save the language file on quit.
2. Map m_ID to our unformated string in a map.

On Sv_LocalizedChat:

• check whether m_ID is mapped

  • if not, send Cl_SetLanguage to notify server that it should clear its memory about what have been sent to us, so it resends Sv_TranslatedFormatString in the future. Abort all following steps and notify user a chat message has been lost.
  • if yes, pull the unformatted message from our map, localize it, and display the localized string in chat.

• The ID to unformatted string map should be cleared on disconnect.

---

For broadcast it is almost the same but server sends three different Sv_TranslatedFormatString if all three lines are used before Sv_LocalizedBroadcast. Then localize the three strings separately. So, broadcast like: Current Weapon: %s\nAmmo: %d/%d\nSomeotherstuff doesn't look awful in the language file and each line can be reused and mish-mashed together.

The reason for three lines only because 0.7 can only display three lines. If a ddnet/0.6 only mods want more lines, they can just put everything in one Sv_LocalizedBroadcast and m_IDLine1, then do the ugly method. (or, preferably, send translated Sv_Broadcast directly so users' language files don't get polluted with too many ugly strings)

---

For multi-language rcon say and broadcast, (e.g. broadcasting tournament announcement in multiple languages). we can fallback to Sv_Say and Sv_Broadcast with some special command that reads a multi-language file and send pretranslated messages, (so they don't pollute users' language file for an announcement that is for a one-time event)

---

Side-effect: Since server can provide translated message and client can save them, translated message become effectively crowdsourced. If a server hoster translated their server message, and a user joined the server, all occurred string will be translated as well on other similar servers & servers with the same strings.

---

Potential abuse: Malicious server can send garbage translation and pollute user's language file. Maybe ask user whether to allow server to provide translations and prompt them to delete all translated message on disconnect if needed.

Also, we can discard translated message by default (so player don't have to click "allow" constantly on join), and let user enable it if they want to. Since ddnet's server string will likely be included in client release already with a few exceptions due to older clients or updated server. The experience in ddnet can be guaranteed, while for other server, manual translation from client-side is the default behavior.

[heinrich5991]

@Kaffeine

I think that's not a good design for translations for DDNet, as it relies on server updates for localization updates when it is rather a UI thing, i.e. it concerns the client. See also #3091.

Sounds like the 0.7 mod unfriendly way. The argument "it relies on %% updates for localization updates" works in both ways. There is a number of users who use old DDNet clients on new servers, and in your case you "rely on the client updates for localization", which is both relies on more updates (there are more clients than servers, instead of one update you're asking every player to update) AND works against the mods.

You can easily update the localization alone in an old client.

What does it have to do with 0.7? I don't think we want to cause effort for 0.7 compatibility.

[Kaffeine]

You can easily update the localization alone in an old client.

I can easily do a lot of things but we're talking about an average/casual players, right? What is the ratio of this "easily" doable thing — how many of old client version users update their translation? I'd expect a zero. If you mean that it is possible to implement client translation auto-updater then we'd still have an issue with delivering that to the old clients.

On the other hand, the last released server side translation will be available for all clients playing on DDNet network. My point is that server-side translation is much easier to deliver to more users.

Sounds like the 0.7 mod unfriendly way.

What does it have to do with 0.7? I don't think we want to cause effort for 0.7 compatibility.

I meant that 0.7 protocol is not mod friendly. It replaces server-side (text) messages by client generated text+sounds+indication, killing the possibility to reuse the events "building blocks" (e.g. use only text or only the sound or only the indication for a different, mod-specific purpose). For example, global sound message was removed and the mods can't use the sound-triggering message (such as flag capture) because the messages also generate texts on the client side).

I see the benefits of UI indicators (e.g. I like https://github.com/ddnet/ddnet/pull/7515) but I don't like the 0.7 way of removing the building blocks from the protocol (you don't seem to remove anything just yet).

[TsFreddie]

~~Email update moment. How is this thread only three years old, feels like an eternity ago~~

Anyway, none of the proposals above prevent server mods sending translations, so I don't see how it was remotely similar to 0.7.

[heinrich5991]

I see the benefits of UI indicators (e.g. I like #7515) but I don't like the 0.7 way of removing the building blocks from the protocol (you don't seem to remove anything just yet).

I also dislike removing these building blocks, but I think client-side translations can be done without removing functionality.

[Bamcane]

(As #9997 was closed, I am re-proposing my suggestion here.) I recommend initiating GameMsg development upon completion of #9549.

After that, I purpose to devide it into two parts so that we can test it in DDNet servers early. Then we can continue to discuss the Mod Translation if the feedback is fine.

[heinrich5991]

I think that focusing on game messages is too little, I think we should rather use a solution that can work for all strings sent from server to client. I.e. including broadcasts, network close messages, etc.