Possibilty to add translations via BGforge's web based translation tool

[brussell1]

I'm currently translating various Fallout 2 mods (UPU,RPU etc) from BGforgeNet to german. The translation is done via Weblate, a web based translation tool, at https://tra.bgforge.net/. After you connect your git repo to it, everyone, who is registered, can help with translating string by string, without caring for the formatting of the source files. Since you ask for help and this project surely gets more popular in the future, I think it could be a good idea to consider this opportunity. Info on how to set this up can be found here: https://forums.bgforge.net/viewtopic.php?f=9&t=26

[fedprod]

While I can't speak for Lexx, I'm personally not a fan of string for string based systems, I need my ressources, a few Notepad++ Windows/Tabs and maybe good music and a drink. That way I can see what I've done and worked on (and bitching about it when one file is waaay too big), but just me beeing old fashioned^^"

Edit: And that's how I finished the translation for 1.1.1946, meaning that readme is not quite up to date ^^ Okay, it can still be better, especially some vanilla stuff and TMAs.

[Lexx2k]

Personally I have no idea how this stuff works and how it is kept up to date.

[burner1024]

Well, the platform is open, if you want to use it, feel free to put an integration request. The system comes with various translation-related perks, but it does have some requirements/limitations as well. The details are on the forum, if something's unclear, feel free to ask too. And if you prefer not to, it's up to you, of course. Whatever works best for translators.

[egornovivan]

So what did you decide? weblate is a very convenient tool for translating text, I see no reason to refuse it.

[Lexx2k]

Not going to happen for this branch. Primary points against it is that 1. you won't be able to directly edit text anymore. Everything needs to be done via that website, and 2. as far as I can tell it works string by string .. so it's possible translators will not understand the context of what exactly they are translating.

Most of the text files in ettu aren't very different from vanilla Fo1. IMO it'll be more convenient to just use a tool like WinMerge and compare vanilla Fo1 with ettu text files and then modify the original text files of language version X to work with ettu.

To give an example, it took me roughly half a day to get most of the german translation up to speed. fedprod did the rest of the files in a couple more days. So all in all it was pretty fast and not a big deal. Hell I didn't even use WinMerge in this case. Just opened every single dialog file and quickly scanned it for things that were obviously different.

[wipe2238]

So. To summary recent IRC talk, translations done on bgforge-tra went from "not going to happen" to "nice to have, if...". Big if being - can we find a working solution to "you won't be able to directly edit text anymore" problem.

For testing, i made a translations branch with script which follows integration instructions and generates .pot/.po files from scratch (based on current state of .msg/.sve/.txt files in master). This is intended to be a base for allowing handmade edits in repo monitored by bgforge-tra. I skipped steps 7+ as i need @burner1024 input before i can continue :p

After reading some of Weblate docs i believe separating .po/.pot is a step in good direction; Weblate will work with its native format only, and EtTu can check translation before it lands into game files (for example by running ScanMsg, as bgforge-tra won't protect mod from broken translations, see BGforgeNet/Fallout2_Restoration_Project#99). If i got every piece right (i'm a total Weblate noob) that's how it's gonna work:

Translation changed on bgforge-tra

• .po file(s) in translations branch are updated as usual
• @antalaskaya runs unpoify (to converts .po files to game format) and ScanMsg
• if everything seems right, master branch is updated (either manually or automagically)

.msg files changed on master branch

• @antalaskaya converts handmade changes to .pot/.po files and commits to translations branch
• webhook informs bgforge-tra about .pot/.po changes
• translators rush to bgforge-tra to translate new string with multiple smiles on their faces

Now the question left is if there's a chance to get following things (bottom of this NMA post makes me little pessimistic about that, but let's bite...):

• using non-default branch for sending/receiving changes all examples i could find uses default branch
• completely disable unpoify commits for everyone that includes translation admins or however they're triggered
• some info about how unpoify commit is actually generated server-side so i can replicate it on my own at the right moment; best if server-side scripts would be included in msg2po repo as well (without any sensitive data, ofc)

[burner1024]

Ahhhh... yes... come, come... join the Dark Side.

First off, cool that you decided to give it a try. It just so happens, however, that right now I'm planning an upgrade to the latest version. It's a big upgrade by itself: 3.1.1 -> 4.7, with intermediate steps, so it's a litlle bumpy. What's making it problematic is that I have to transfer/recreate the customizations added back then to handle female translations. It should be less hacky now with new functionality, but still takes time. msg2po utils will have to be adjusted accordingly too. And while working on that, I discovered a blocker in the said functionality, which prevents me from proceeding with upgrade.

So for now the best thing to do is just sit tight. First I need a fix from upstream, then I'll do the upgrade and add customizations, then probably a week or so to stabilize things.

---

To answer your questions and raise some other random points (gonna be a long list, so I guess I'll numerate for easier reference):

1. Manual edits. Yes, you can edit the texts manually and then update po files manually. I never meant it otherwise. It's simply that if there's one source of truth, everything goes from it. If there's more than one, then someone has to keep them synced manually anyway, resolve conflicts, etc. Which really goes for anything, not just this system. So, to me it's extra work for no gain.
2. And when you do manual syncing, your continuous localization is... less continuous. Obviously it matters less in projects where source changes rarely, but still latency adds up.
3. Integration instruction you linked is for WeiDU mods, for Fallout it's not so strict, less steps. (Basically, can just look at UPU repo). You do want to lowercase dialog files, or at least make sure to keep the case consistent, otherwise you'll be looking at a lot of diff noise.
4. Checks. True, the system doesn't have built-in tests for msg file format. It was never the goal. Still, using msg2po utils gives pretty safe output. If I remember correctly, in that case the translations weren't broken as in "game is broken", gameplay wise it's just strings with typos.
   1. It can probably be made even more safe by escaping/removing/replacing curly brackets in unpoify.
   2. Weblate has its own list of checks that can be applied, and it includes auto fixups, and custom ones can be added too, see the same doc. It is a long list, though, and can become overbearing if used without caution.
   3. In the end, no tool can find all typos, but it's good to have automatic tools to run checks for common issues. Even better when they overlap and cross-check each other.
   4. I tried ScanMsg for UPU, I think it was throwing warnings on some english source strings, not allowing {000} strings or something, and I didn't want to delete them from the source, because they were actually useful comments. Didn't find how to get around that, so disabled the action for the time being.
5. Custom branches are possible, weblate itself uses a non-master branch. You can also do pull requests and stuff. I don't do either in my repos, because
   • don't really see the need
   • see point 2)
   • uniform component configuration is much easier to maintain
6. Unpoify
   1. Not sure who do you mean by "everyone/translation admins". If you mean weblate admins, so far there's one - me.
   2. But no one runs unpoify manually. It is launched by a pre-commit hook.
   3. Hook can be disabled.
   4. What it does literally is unpoify && git add ., that's all.
   5. I actually thought about making poify/unpoify into github actions. On one hand, it's going to take some load off the server. On the other, that's extra configuration steps for every repo (and since most repos now are mine, that means many more steps for me). Plus msg2po releases will have to be handled more carefully. Plus it's extra vendor lock on github/microsoft. So... undecided.
7. Extra random points you may or may not know yet, but pointing them out just in case so that I don't get accussed of ruining everything later :):
   1. PO format stuffs all identical strings into a single translation unit, which reduces workload, speeds up translation and ensures far better consistency. However, there are edge cases where that is undesirable, and you have to resort to workarounds, see 1, 2, 3. (Practically, I had 2-3 cases of this in several years)
   2. bgforge-config.py can be used to get values from .bgforge.yml
   3. dir2msgstr which I see you using is a crude tool. It was created as a quick and dirty way to load existing old translations into empty POs once and use that as a base for continuous translation. Not meant for continuous loading, you're on your own if things break. Didn't look at it for a long time, but in particular, I'm pretty sure it doesn't have the smarts to handle female translations like unpoify does.
   4. Speaking of female translations, you know about those, right? So it's not 2 sources of truth to keep in sync, but 3 at least.
   5. While uniformity goes a long way, you don't have to use the system for all languages. If you want to keep translating, say, German manually, you can just not generate a po for it and translate it manually like before. You can add languages gradually. Start with Polish or something.
   6. Holodisk entries aren't great for unit translation, because you need to align them to display well, so recently I skipped pipboy.msg.

[wipe2238]

Integration instruction you linked is for WeiDU mods, for Fallout it's not so strict, less steps. (Basically, can just look at UPU repo) dir2msgstr (...) doesn't have the smarts to handle female translations (..._ you're on your own if things break (...)

In short, there's no way for me to even start with bgforge-tra now ( ._.)

Fallout instructions doesn't exist (plus i have no idea how UPU can help me, as it contains already baked stuff; same for any other Fallout translations repo), i can't use only msg2po tool which actually created language.po (on your own = don't ever touch that for me, as i don't even use/know python to fix anything), i can't use other msg2po tools due to female translations stuff, the list goes on.

Maybe current weather and daily 30'+ made me blind dum-dum, but i simply have no idea how i'm supposed to go with initial preparation of language.po files now, not to mention updating them in future... I really hope that incoming update can address that in some way, as i could really use a tool which prepares language.po in a single command with female translations and gods knows what else is going on under the hood (now and in future). Maybe something like... poify --template english.pot some/language/ which would generate language.pot in memory and then merge it with english.pot into language.po, while respecting english.pot files locations (drop files not present in base language, include files missing in translation dir, etc).

You do want to lowercase dialog files, or at least make sure to keep the case consistent

I'll go with second route (NAME.msg) then; had it in backlog for a long time anyway, for unrelated reasons.

system doesn't have built-in tests for msg file format. It was never the goal.

Just to be clear, i'm not saying that's a flaw in bgforge-tra. We simply had waaay too much crashes/reports in past caused by little tiny mistakes in .msg files, that we're freaking paranoid now and have zero trust policy to any tool touching them - including our own :S

ScanMsg (...) not allowing {000} strings or something

Either it was some old version (where i wrongly assumed id 0 is not valid), or a newer one complained about duplicated id (as # does not make {} invisible to engine, same usually applies to ScanMsg); you can:

• replace {} with something else (whitespace, (), etc) making it a real comment
• replace each {000} with some unique id not used ingame ({991}, {992}, and so on)

This specific case must be fixed in .msg; any change application-side would affect real entries which are more important than decades old Interplay comment, even as useful as this one.

Not sure who do you mean by "everyone/translation admins". If you mean weblate admins, so far there's one - me. But no one runs unpoify manually

OK, thanks for clarifying how it works.

I actually thought about making poify/unpoify into github actions

You could avoid any vendor lock by using git hook which can also be ran as standalone script (as in: can do its job without using any switches/env vars provided by git). Actions (but also other CI using similar mechanic) could serve as sanity check layer which, in the end, calls the mentioned script with proper arguments if everything is in place. If other CI does not offer such solution, or you want to run manually/as git hook, you still have script in place.

female translations, you know about those, right?

Yep. I'll worry about that when we're close to accepting .po commits, it's too distant future for now.

[burner1024]

i simply have no idea how i'm supposed to go with initial preparation of language.po files now, not to mention updating them in future... I really hope that incoming update can address that in some way, as i could really use a tool which prepares language.po in a single command with female translations and gods knows what else is going on under the hood (now and in future).

Initial PO is a copy of POT with strings loaded by dir2msgstr. POT is generated by poify. The only caveat is to make sure that translation being loaded matches the original language files unit for unit. As far as I understand, you don't have any female translations, so you don't need to worry about them just yet. Added Fallout guide. Note that you don't have to generate PO files yourself, I can do that on initial setup. (But I'll do that only once per mod). Of course, you can do that yourself too - less work for me!

"On your own" specifically refers to using ...2msgstr tools repeatedly to load manually changed strings into POs. They are meant to be used once, for initial load. I'm not sure how well they will work in repeating scenario, and I think it's a bad practice anyway, so I don't plan to spend time on that now, nor after upgrade. But, these are simple enough tools, and I'm sure that if you wanted, you could easily re-create them in your language of choice. Basically, it's just parsing files with regex, and then adding found matches into PO.

We simply had waaay too much crashes/reports in past caused by little tiny mistakes in .msg files, that we're freaking paranoid now and have zero trust policy to any tool touching them - including our own :S

I totally understand, though I do find it peculiar with you had crashes caused by mistakes made by humans, yet you're paranoid about tools. I, on the other hand, so far had zero crashes reported that were due to bad msg files, so I think the policy of not touching msg by hand is working out pretty well - just saying. When Nevada or Sonora, or some other translation update is posted, it's quite often followed up by a hotfix, because someone screwed up the brackets. And every year or two, yet another tool for msg checking pops up... I just laugh in Python when I see these.

[burner1024]

So I made some actions. Feel free to try them out, example here. Suggestions for improvements are welcome, pull requests moreso. I'm not very familiar with gha creation. Not sure if I should combine them...

[wipe2238]

Won't be able to test/mess with shiny stuff until next week/next weekend, so for now just a quick look without running:

---

- run: |
       set -xeu -o pipefail
       action_root="$(realpath $(dirname $(dirname ${{ github.action_path }})))"
       export PATH="$PATH:$action_root"
       ${{ github.action_path }}/poify.sh

• It's recommended to use GitHub's way of updating PATH when possible
• PATH will be updated when current step finishes
• Actions as they are right now will fail when running on Windows, as github.action_path will use Windows-style directory separators which bash doesn't really like. Lucky, there is env copy of it ($GITHUB_ACTION_PATH, set only during composite action execution) so it can be quickpatched with bash (it's know issue but GitHub decided to ignore that; actions/runner#1066)

With all of that in mind, i'd go with something like this; maybe wrap it in $(realpath ...) for a good measure :p

- run: |
       set -xeu -o pipefail
       echo "${GITHUB_ACTION_PATH//\\//}../../" >> $GITHUB_PATH
- run: |
       set -xeu -o pipefail
       ${GITHUB_ACTION_PATH//\\//}/poify.sh

---

Kinda edge case from your perspective, but it would be nice if commit+push could be made optional. That way user have more control about who and when commits/pushes to repository if for some reason pushing right away isn't desired for some reason (for example in EtTu i allow only 1 automated commit+push to master per day (to not annoy Lexx when he's in crunch mode :p), unless it's called manually with specific option enabled. If un/poify would allow skipping commit+push, i could include results in single commit, next to all other stuff we already have there).

[burner1024]

If un/poify would allow skipping commit+push, i could include results in single commit, next to all other stuff we already have there).

But it's literally the only thing that it does. If you don't want a commit, just don't run it? Or what all other stuff do you mean?

Edit: anyway, I combined the actions and added some options. Edit2: technically, it should even be possible to allow translations flow both ways, from web system and from direct edits, though of course it'll require some scripting and choosing which edits take precedence in case of conflicts. Edit3: it seems that upstream fix is being delayed. I guess we can proceed with current version if you want.

[egornovivan]

@wipe2238 Will you wait or continue with the integration?

[burner1024]

leaving here for future reference https://stackoverflow.com/questions/16214067/wheres-the-3-way-git-merge-driver-for-po-gettext-files

[burner1024]

Just fyi, it's almost all done. (edit: all done) I got a little carried away, so now dir2msgstr should support repeated runs. (Although I still don't like it and don't recommend it.)