Windows-built hledger executable fails in WSL

[ShrykeWindgrace]

Hi!

I've encountered the following problem with character encoding.

My setup:

• Win 10 with an Ubuntu18 in WSL
• hledger is built from source on the windows side:

bash $ hledger.exe --version
hledger 1.16.1

• locale in Ubuntu:

bash $ locale
LANG=en_US.UTF-8
LANGUAGE=
LC_CTYPE="en_US.UTF-8"
LC_NUMERIC="en_US.UTF-8"
LC_TIME="en_US.UTF-8"
LC_COLLATE="en_US.UTF-8"
LC_MONETARY="en_US.UTF-8"
LC_MESSAGES="en_US.UTF-8"
LC_PAPER="en_US.UTF-8"
LC_NAME="en_US.UTF-8"
LC_ADDRESS="en_US.UTF-8"
LC_TELEPHONE="en_US.UTF-8"
LC_MEASUREMENT="en_US.UTF-8"
LC_IDENTIFICATION="en_US.UTF-8"
LC_ALL

• UTF-8 encoded journal

• run hledger.exe -f journal register inside WSL (launched via bash/conemu/ubuntu, no matter) to get

bash $ hledger.exe -f journal register
hledger.exe: E:\nosave\bookkeeping\journal: hGetContents: invalid argument (invalid byte sequence)

Yet on the windows side in a unicode-aware (via chcp 65001) powershell I get the correct output.

As suggested in the "troubleshooting" section in the manual, I tried running LANG=something hledger -f journal register; it does not help either for reasonable choice of "something".

What else can I try in this situation? Try to use an ubuntu-in-wsl-built version of hledger, maybe?

[simonmichael]

Yes, that should definitely work. It's unfortunate that your binary almost works fine in both environments. I don't know if it's something easy or impossible to fix.. but I remember from #961 (found on download page) the conclusion was "if you want non-ascii handled correctly, don't run hledger in a Windows environment (cmd, msys...) different from the one that built it".

[simonmichael]

PS if you think of improvements to the Troubleshooting section or download page, they are welcome.

[ShrykeWindgrace]

Well, I would not say that they almost work, only WSL gives my troubles - my powershell is always set to chcp 65001 for plenty of other reasons, so hledger works there just fine.

It just seems weird to maintain a herd of terminals (ps, cmd, bash, git bash, mobaXterm,...) on windows because each can do something that others can't, and then again maintain a copy of a binary for each terminal because they all handle unicode/something else differently (rant mode off =) )

I'll test a wsl-built hledger later this evening and report my results

[simonmichael]

"Almost" meaning the hledger binary works.. except for non-ascii data. Yes it is weird, and unfortunate.

[ShrykeWindgrace]

I did a little bit of debugging, it seems one of the culprits is the file enconding detection. The script goes like this (inspired by https://github.com/simonmichael/hledger/blob/65efdea4c04913caa83f765da1f68d4157d9430d/hledger-lib/Hledger/Utils.hs#L203), built in unicode-aware powershell.

module Main where

import System.IO
import System.Environment
main :: IO ()
main = do
  [f] <- getArgs
  withFile f  ReadMode penc

penc :: Handle -> IO ()
penc h = hGetEncoding h >>= print

and I ran it in different environments on the same file; I know for sure that this file is utf-8 with cyrillic characters.

• powershell with $OutputEncoding = [console]::InputEncoding = [console]::OutputEncoding = New-Object System.Text.UTF8Encoding gave "UTF-8" (both standalone and in ConEmu)
• git bash "as is" gave "CP437" (I tried to fiddle with LC_ALL, LANG, and LANGUAGE to no avail). file gives "UTF-8" under the same conditions
• git bash run via ConEmu surprisingly gave UTF-8 (wtf???)
• cmd ("as is" and in ConEmu) gave "CP437"
• cmd ("as is" and in ConEmu, but after chcp 65001) gave "UTF-8"
• bash in WSL (both via ConEmu or "as is") gave "CP1252", fiddling with env vars gave nothing. file utility correctly reports "UTF-8".

After this investigation I am inclined to start googling for a robust file encoding detection in Haskell, because clearly delegating this task to the host OS and terminal gave wildly inaccurate results. Or add an option "--encoding=utf8" to hledger, but this is a more daunting undertaking.

[Seriousness]

I actually put up my wsl (with arch apparently) again just to be able to handle hledger in a utf8 environment on my business laptop and it works perfectly. Files are still iny my windows environment...

[simonmichael]

@Seriousness, good to know. Any special know-how needed for doing that, for other Windows users ?

[Seriousness]

for me it was only getting the package from pacman... guess it should be the same with apt. Got myself a link (ln -s sourcedir linkname) to get quickly to the folder after starting wsl (cd to the link instead of typing the whole folder structure)

ah yeah, I work with the -f parameter all the time, but it should work with the userdir .hledger, too

edit:// its not that easy, hardlinks wont work as wsl understands the /mnt/* as different partitions. include does not work. Won't find the file. My guess: Its not understanding the root "/"?

[simonmichael]

Reopened for confirmation.

[ShrykeWindgrace]

@simonmichael I will able to test it in June - at least I hope so=)

[simonmichael]

@ShrykeWindgrace: hi, do you know if this is still unresolved ?

[ShrykeWindgrace]

Whoa, that's quite a delay from me, I promised to answer a year ago. Sorry=)

Setup:

• Win 10, Windows Terminal, powershell-7
• hledger built over a nightly lts in February: hledger 1.28-g40306a6ae-20221201, windows-x86_64
• WSL1 (ubuntu-20) launched in Windows Terminal as host; hledger.exe works just fine for utf8-encoded files.

That being said, I no longer use other terminals (Windows Terminal is sufficient for my needs right now). For shells - only powershell-7 and bash in WSL1. Moreover, I no longer run windows executables from WSL either, I use whatever version available in nixpkgs-unstable.

In the current state, everything is ok for me, so feel free to close the ticket. If you need to further testing, I can do that, too.

Cheers!