[Bug]: Mojibake in Japanese games

[human222]

Describe the bug

Language is set to Japanese within Bottles, so I would expect Japanese text to work out of the box. Instead I get garbled text (mojibake).

To Reproduce

1. Create and/or open a bottle. I used the gaming preset.
2. Go to Settings -> Compatibility -> Language and set it to Japanese
3. Go back and click "Run executable..." and run a game.

Package

Flatpak from Flathub

Distribution

Debian Bookworm

Debugging Information

Official Package: true
Version: '50.2'
DE/WM: gnome
Display:
    X.org: true
    X.org (port): :0
    Wayland: true
Graphics:
    vendors:
        amd:
            vendor: amd
            envs:
                DRI_PRIME: '1'
            icd: /usr/lib/x86_64-linux-gnu/GL/vulkan/icd.d/radeon_icd.x86_64.json:/usr/lib/i386-linux-gnu/GL/vulkan/icd.d/radeon_icd.i686.json
    prime:
        integrated: null
        discrete: null
Kernel:
    Type: Linux
    Version: 6.1.8-x64v2-xanmod1
Disk:
    Total: 8351539200
    Free: 8351391744
RAM:
    MemTotal: 15.6GiB
    MemAvailable: 11.9GiB
Bottles_envs: null

Troubleshooting Logs

No response

Additional context

No response

[StoneMoe]

Hi, Could you try:

1. set Language option to "System" if your host locale is already set to Japanese
2. Install cjkfonts from Dependency List

[human222]

My host locale isn't set to Japanese. Attempting to install cjkfonts only revealed another bug. I also could not find a crash report after restarting Bottles. The terminal output doesn't seem to help either:

flatpak run com.usebottles.bottles 05:00:42 (INFO) Bottles Started! 05:00:43 (INFO) Performing Bottles checks… 05:00:43 (INFO) Dxvks found: - dxvk-2.1

05:00:43 (INFO) Vkd3ds found: - vkd3d-proton-2.8-1-1e663e6

05:00:43 (INFO) Nvapis found: - dxvk-nvapi-v0.6-1-0827efb

05:00:43 (INFO) Latencyflexs found: - latencyflex-v0.1.1

05:00:43 (INFO) Runners found: - sys-wine-7.0.1 - soda-7.0-8

05:00:43 (INFO) Bottles found: - default

(bottles:2): Gtk-CRITICAL **: 05:00:43.726: gtk_widget_set_parent: assertion '_gtk_widget_get_parent (widget) == NULL' failed 05:00:43 (INFO) Library saved 05:00:44 (INFO) Catalog installers loaded 05:00:44 (INFO) Catalog dependencies loaded 05:00:44 (INFO) Catalog components loaded 05:00:45 (INFO) Setting Key sync=fsync for bottle default… 05:00:45 (INFO) There is no running wineserver. 05:00:54 (INFO) Installing dependency [cjkfonts] in bottle [default]. 05:00:54 (WARNING) File [SourceHanSans.ttc.zip] already exists in temp, skipping. patool: Extracting /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip ... patool: running /app/bin/7z x -o/home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc -- /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip patool: ... /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip extracted to `/home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc'. patool: Extracting /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc/SourceHanSans.ttc ...

[StoneMoe]

My host locale isn't set to Japanese. Attempting to install cjkfonts only revealed another bug. I also could not find a crash report after restarting Bottles.

this installation error is fixed in e247addd6f420ee1353c45f57b9dced213885326 but not released yet. meanwhile you can install latest fixed built Artifact from https://github.com/bottlesdevs/Bottles/actions/runs/4019870533

[human222]

Trying the linked artifact, I'm getting a different terminal output:

flatpak run com.usebottles.bottles 05:22:29 (INFO) Bottles Started! 05:22:30 (INFO) Performing Bottles checks… 05:22:30 (INFO) Dxvks found: - dxvk-2.1

05:22:30 (INFO) Vkd3ds found: - vkd3d-proton-2.8-1-1e663e6

05:22:30 (INFO) Nvapis found: - dxvk-nvapi-v0.6-1-0827efb

05:22:30 (INFO) Latencyflexs found: - latencyflex-v0.1.1

05:22:30 (INFO) Runners found: - sys-wine-7.0.1 - soda-7.0-8

05:22:30 (INFO) Bottles found: - default

05:22:30 (INFO) Catalog installers loaded 05:22:30 (INFO) Catalog dependencies loaded 05:22:30 (INFO) Catalog components loaded 05:22:34 (INFO) Setting Key sync=fsync for bottle default… 05:22:34 (INFO) There is no running wineserver. 05:22:45 (INFO) Installing dependency [cjkfonts] in bottle [default]. 05:22:46 (WARNING) File [SourceHanSans.ttc.zip] already exists in temp, skipping. patool: Extracting /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip ... patool: running /app/bin/7z x -o/home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc -- /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip patool: ... /home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc.zip extracted to `/home/segphault/.var/app/com.usebottles.bottles/data/bottles/temp/SourceHanSans.ttc'. 05:22:47 (INFO) Adding Key: [HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\Fonts] with Value: [Source Han Sans SC ExtraLight] and Data: [SourceHanSans.ttc] in default registry 05:22:47 (INFO) Using Wine Registry CLI -- add /bin/sh: warning: setlocale: LC_ALL: cannot change locale (ja_JP) fsync: up and running. wine: RLIMIT_NICE is <= 20, unable to use setpriority safely 002c:err:wineboot:process_run_key Error running cmd L"C:\windows\system32\winemenubuilder.exe -r" (2). 05:22:47 (ERROR) Error while running async job: <bound method DependencyManager.install of <bottles.backend.managers.dependency.DependencyManager object at 0x7f7fb6d15b10>> Exception: a bytes-like object is required, not 'str'

File "/app/share/bottles/bottles/frontend/utils/threading.py", line 61, in __target result = self.task_func(*args, **kwargs) File "/app/share/bottles/bottles/backend/managers/dependency.py", line 148, in install res = self.__perform_steps(config, step) File "/app/share/bottles/bottles/backend/managers/dependency.py", line 278, in __perform_steps self.__step_register_font( File "/app/share/bottles/bottles/backend/managers/dependency.py", line 658, in __step_register_font reg.add( File "/app/share/bottles/bottles/backend/wine/reg.py", line 79, in add logging.info(res, ) File "/app/share/bottles/bottles/backend/logger.py", line 69, in info self.root.info(self.__color("info", message), ) File "/app/share/bottles/bottles/backend/logger.py", line 48, in __color if message is not None and "\n" in message:

[StoneMoe]

@human222 unfortunately it's revealing another bug again...

Do you have any Japanese font installed in your host system? if you do, the cjkfonts should not be necessary

[human222]

I do, and Japanese text works fine in Linux applications.

[StoneMoe]

What program are you trying to run exactly? Could you provide it? so we can run more tests against it.

[human222]

Now Bottles just refuses to run any Windows program. It seems to get stuck with this:

002c:err:wineboot:process_run_key Error running cmd L"C:\\windows\\system32\\winemenubuilder.exe -r" (126).

There already seems to be a bug report for this with an unknown solution. As for what program, the issue should show up in pretty much any Japanese program. As an example, I think "Ys VI - Napishtim no Hako (Japan)" had the mojibake issue. I can't continue testing as Bottles is completely broken.

[StoneMoe]

winemenubuilder.exe is blacklisted by default in Bottles, so it should be fine with this particular error message.

I can run Task Manager with Japanese language without mojibake issue Could you try Task Manager or any builtin tools too?

[human222]

I tested some builtin tools. Explorer's interface seems ok, but it can't display any folders with a Japanese name. Bottles probably doesn't set the correct locale:

Task manager looks ok at first:

But here text is getting cut off:

wineconf looks mostly ok but there is some mojibake in the DPI section:

I'm guessing that some builtin Wine applications are standard UTF-8 in some places but not everywhere. Real-world (Japanese) Windows applications also tend to use a Japanese encoding. There also seem to be issues with assigning fonts with appropriate proportions.

[StoneMoe]

yes, Japanese programs are usually using shift-jis or euc-jp, and shift-jis is the most popular encoding since windows is using cp932(variant of shift-jis) AFAIK

unfortunately neither Linux nor flatpak runtime provide shift-jis encoding by default so there is no easy fix for this issue currently. Still, I'll keep trying to find a way to fix this, a minimally reproducible example would be a big help.

also you can try tools like https://github.com/xupefei/Locale-Emulator to see if it will work under Wine, and following this guide to see if it works with a bare wine without flatpak

[CKolle]

This issue seems to come down to that bottles sets LC_ALL=ja_JP when selecting Japanese. When it should set it to LC_ALL=ja_JP.UTF-8. This is what proton and lutris sets when changing the locale/language to Japanese. They don't have this issue where the paths are interpreted as euc-jp. I have also tested by setting "LC_ALL=ja_JP.UTF-8", and I had no issues with explorer as it could interpret my UTF-8 paths now. Also tried opening a file with shift-jis encoding and it had no problem with that using LC_ALL.UTF-8 (using both soda 8 in bottles and wine 9 outside of bottles). According to a wine testing page "UTF-8 is a character set suffix, and all current Linux distributions use UTF-8 by default" so I think a sane default would be to set it to ja_JP.UTF-8 instead of ja_JP.