Unicode Symbols replaced with `?` on Windows

[scottmckendry]

Neovim version (nvim -v)

v0.10.0-dev-2098+g92672a161

Operating system/version

Windows 11 23H2

Add the debug logs

• [X] I have set log_level = vim.log.levels.DEBUG and pasted the log contents below.

Log file

15:54:22[DEBUG] Running formatters on C:\Users\scott\git\windots\test.ps1: { "powershell" } 15:54:22[INFO] Run powershell on C:\Users\scott\git\windots\test.ps1 15:54:22[TRACE] Input lines: { '$emoji = "👍"', '$folderIconNerdFont = ""' } 15:54:22[DEBUG] Run command: { "pwsh", "-NoLogo", "-NoProfile", "-NonInteractive", "-Command", "(Invoke-Formatter", "(Get-Content -Raw -Path", "C:\Users\scott\git\windots\test.ps1", ")).Trim()" } 15:54:23[DEBUG] powershell exited with code 0 15:54:23[TRACE] Output lines: { '$emoji = "??"\r', '$folderIconNerdFont = "?"\r' } 15:54:23[TRACE] powershell stderr: { "" } 15:54:23[TRACE] Applying formatting to C:\Users\scott\git\windots\test.ps1 15:54:23[TRACE] Comparing lines { '$emoji = "👍"', '$folderIconNerdFont = ""' } and { '$emoji = "??"\r', '$folderIconNerdFont = "?"\r' } 15:54:23[TRACE] Diff indices { { 1, 2, 1, 2 } } 15:54:23[TRACE] Applying text edits: { { newText = '??"\r\n$folderIconNerdFont = "?"\r', range = { ["end"] = { character = 27, line = 1 }, start = { character = 10, line = 0 } } } } 15:54:23[TRACE] Done formatting C:\Users\scott\git\windots\test.ps1

Describe the bug

This is isolated to Windows only. I've tested this on Linux with the same config and format command and it works perfectly.

Unicode chars like emoji and icons get replaced with ?. At present, I can only replicate this with PowerShell but wouldn't be surprised if other formatters are affected.

Before:

After:

Running the format command directly from the command line works fine. So I can only assume there is something weird about how Conform is handling the output.

What is the severity of this bug?

breaking (some functionality is broken)

Steps To Reproduce

1. nvim -u minimal.lua
2. :e test.ps1
3. enter emojis, nerd font icons etc.
4. save

Expected Behavior

Characters should not change during formatting.

Minimal example file

$emoji = "👍"
$folderIconNerdFont = ""

Minimal init.lua

-- DO NOT change the paths and don't remove the colorscheme
local root = vim.fn.fnamemodify("./.repro", ":p")

-- set stdpaths to use .repro
for _, name in ipairs({ "config", "data", "state", "cache" }) do
    vim.env[("XDG_%s_HOME"):format(name:upper())] = root .. "/" .. name
end

-- bootstrap lazy
local lazypath = root .. "/plugins/lazy.nvim"
if not vim.loop.fs_stat(lazypath) then
    vim.fn.system({
        "git",
        "clone",
        "--filter=blob:none",
        "--single-branch",
        "https://github.com/folke/lazy.nvim.git",
        lazypath,
    })
end
vim.opt.runtimepath:prepend(lazypath)

-- install plugins
local plugins = {
    "folke/tokyonight.nvim",
    {
        "stevearc/conform.nvim",
        config = function()
            require("conform").setup({
                log_level = vim.log.levels.DEBUG,
                -- add your config here
                formatters_by_ft = {
                    ps1 = { "powershell" },
                },

                format_after_save = {
                    lsp_fallback = true,
                },

                formatters = {
                    powershell = {
                        command = "pwsh",
                        args = {
                            "-NoLogo",
                            "-NoProfile",
                            "-NonInteractive",
                            "-Command",
                            "(Invoke-Formatter",
                            "(Get-Content -Raw -Path",
                            "$FILENAME",
                            ")).Trim()",
                        },
                    },
                },
            })
        end,
    },
    -- add any other plugins here
}
require("lazy").setup(plugins, {
    root = root .. "/plugins",
})

vim.cmd.colorscheme("tokyonight")
-- add anything else here

Additional context

No response

[scottmckendry]

For some clarity, when testing on formatting cmd on its own, I'm running pwsh -NoProfile -NoLogo -Command "(Invoke-Formatter (Get-Content -Raw -Path ./test.ps1)).Trim()"

Which writes the file to stdout as expected:

[c00t]

conform.nvim could be calling pwsh from cmd? And then reading stdout? So your script example doesn't prove anything (pwsh can handle unicode well). This is most likely a cmd problem?

I just solved a encoding problem on windows due to a cmd AutoRun script...

code ref: https://github.com/stevearc/conform.nvim/blob/192a6d2ddace343f1840a8f72efe2315bd392243/lua/conform/runner.lua#L322-L334

you can just log data in on_stdout to track stdout by your command.

---

emm, I believe vim.fn.jobstart uses platform specific shell .

just thought of another possibility, vim.fn.jobstart use a different profile(so different shell encoding) than the one used for manually calling pwsh

[GitMurf]

conform.nvim could be calling pwsh from cmd? And then reading stdout? So your script example doesn't prove anything (pwsh can handle unicode well). This is most likely a cmd problem?

I believe this is likely the issue. The default for Neovim on Windows is to go through cmd. Even if invoking powershell, it is still cmd invoking powershell... cmd is the middleman between neovim and the powershell process.

As a side note, is there a reason @scottmckendry that you want to use powershell invoke-formatter instead of using the powershell LSP formatter?

[scottmckendry]

I've never been able to get LSP formatting working with PowerShell. Not sure what it is but for whatever reason, LSP fallback just isn't working with my PSES config.

Which is weird, because vim.lsp.buf.format works just fine. But you're right, it doesn't have the same problem with emojis etc. As a happy medium, I've got this as a workaround in my config:

format_after_save = function()
    if vim.g.disable_autoformat then
        return
    else
        if vim.bo.filetype == "ps1" then
            vim.lsp.buf.format({ async = true })
            return
        end
        return { lsp_format = "fallback" }
    end
end,

Which seems to do the trick! Thanks for the steer @GitMurf 🙂

[GitMurf]

@scottmckendry here is my config... notice I am using format "on" save and you are using format "after" save.

I allow for fallback to lsp as long as it is not js / ts because those are really the only ones that I use a non lsp formatter (prettier). So everything else I am good with using the lsp fallback.

One thing to note that could have created problems for you in the past (assuming maybe you tried format "on" save in the past) is that you cannot set to async. I had a specific comment in my config (see below) which leads me to believe I had a problem in the past trying to use async. So maybe try using format "on" save like mine below and make sure not to use async and see if it works 🤷‍♂️

EDIT: also notice I am setting quite a high timeout at 3 seconds. I have some large files that sometimes prettier can take a bit to format so I set it high to make sure I always have enough time to successfully format. You may want to experiment with that as well and set a high timeout just in case (to see if you can at least get working).

    format_on_save = function(bufnr)
      if vim.g.disable_autoformat or vim.b[bufnr].disable_autoformat then
        return
      end
      -- Disable "format_on_save lsp_fallback" for languages that have a formatter
      local disable_filetypes = {
        javascript = true,
        javascriptreact = true,
        typescript = true,
        typescriptreact = true,
      }
      return {
        timeout_ms = 3000,
        lsp_fallback = not disable_filetypes[vim.bo[bufnr].filetype],

        -- NOTE: format_on_save cannot use async = true
        -- async = true,
      }
    end,

[GitMurf]

Also make sure you remove powershell from the conform formatters list since there is not one and you want to use the lsp fallback.