Problems with encoding UTF-8 in Powershell

[ingfors]

Per default Windows 10 and Windows 11 do not use UTF-8 unless you manually enable as a Beta feature under Region Settings in intl.cpl

Without that setting enabled both Sublime and Powershell are having problems reading and displaying special characters such as "åäö".

Here is an example below on a machine without UTF-8 enabled running files from the Powershell console.

File encoding: UTF-8

File encoding: UTF-8 with BOM

As you can see Powershell is unable read the string when the file encoding is "UTF-8" and everything works fine when the encoding is set to "UTF-8 with BOM".

Here is another example with the Build Panel is Sublime Text File encoding: UTF-8

File encoding: UTF-8 with BOM

As you can see the output is still wrong in the Build Panel even when the encoding is set to "UTF-8 with BOM". But it can be fixed by running chcp.exe in cmd

And then changing the encoding settings in the build job.

As stated before this is only an issue when the UTF-8 Beta feature in Windows is disabled. If you enable UTF-8 support the Sublime Text build panel and powershell console works without any issues.

I am not sure how to handle the build panel problem but at least changing the default encoding format for Powershell files to "UTF-8 with BOM" would prevent scripts processing special characters from causing unwanted bugs. This could probably be done by updating PowershellSyntax.sublime-settings with "default_encoding": "UTF-8 with BOM". But I am not sure how that would affect Linux or Powershell Core users.

[deathaxe]

LaTeXTools implements an automatic encoding guesser to ensure correct display.

https://github.com/SublimeText/LaTeXTools/blob/fc0d1791988d0bbbd5e7d96dc390aca06f81f37a/latextools/make_pdf.py#L50-L76

It would however require to provide a dedicated build runner extending Default/exec.py.

[deathaxe]

Even easier solution: set "encoding": "oem".

{
    "cmd": ["pwsh", "-nologo", "-noprofile", "-ExecutionPolicy", "Bypass", "-file", "$file"],
    "selector": "source.powershell",

    "windows": {
        "encoding": "oem",
    }
}

[michaelblyons]

Should we just set that in the Windows variant, then?

[deathaxe]

It seem so be the simples and most straight forward default.

I haven't checked what happens if utf-8 output is enabled via Control Panel though.

At least it would cause malformed output, if chcp 65001 is called in a script.

Maybe provide it as a default and add another "unicode / utf-8" variant.

[beautifulcoder]

I am also seeing this issue with the Build Panel in PowerShell 7, the output shows up as garbage in Sublime. Changing the encoding does not fix the issue, I have to resort to powershell which the older version instead of pwsh which is the latest.

[beautifulcoder]

Verified this same encoding issue happens in Linux with the pwsh command.

[michaelblyons]

@beautifulcoder That looks like ANSI escape sequences, which is a different problem from the original complaint.

A bit of reading suggests that you might look into setting $PSStyle.OutputRendering to PlainText.

Maybe "TERM": "dumb" in an "env" key would work?

[michaelblyons]

@beautifulcoder Try #193 and see if that works for you.