jbang on PowerShell handles non-ASCII characters in home directory badly

[laszlovandenhoek]

trafficstars

Describe the bug If your Windows home directory contains "special" characters, you need to manually fix your PATH variable after the Powershell installer finishes. Also, you must change the default cache folder (by setting JBANG_CACHE_DIR), and if you're using dependencies also the default repo folder (by setting JBANG_REPO), or jbang won't run any code.

To Reproduce

1. Create a new Windows user with a username that contains non-ASCII characters. Example: LászlóvandenHoek
2. Log in as that user and attempt to install JBang in a PowerShell window: iex "& { $(iwr https://ps.jbang.dev) } app setup" (as per https://www.jbang.dev/download/)
3. Close and open new Powershell window as instructed. Attempt to use jbang. Doesn't work, command not found.
4. Inspect the new PATH variable, notice it has been prepended with: C:\Users\Lßszl≤vandenHoek\.jbang\bin instead of the correct C:\Users\LászlóvandenHoek\.jbang\bin.
5. Correct the path manually, start a new PS window
6. Run jbang - this works
7. Run jbang --verbose https://gist.github.com/maxandersen/f43b4c52dfcfc42dcd59a04e49acf6ec - This does NOT work, output:

[jbang] jbang version 0.83.1
[jbang] Gist url api: https://api.github.com/gists/f43b4c52dfcfc42dcd59a04e49acf6ec
[jbang] Retrieved file from cache https://gist.githubusercontent.com/maxandersen/f43b4c52dfcfc42dcd59a04e49acf6ec/raw/a1078d0f3c3f5a6cb5faf82e487faec484bf6be3/fizzbuzz.java = C:\Users\LászlóvandenHoek\.jbang\cache\urls\bf93e83346108430627aeabc2aefbf7d6a02ec82548418bd90f363691c31d157\fizzbuzz.java
[jbang] System Java version detected as 16
[jbang] System Java version matches requested version 16
[jbang] run: 'C:\Program Files\AdoptOpenJDK\jdk-16.0.1.9-hotspot\bin\java.exe' -classpath 'C:\Users\Lßszl≤vandenHoek\.jbang\cache\jars\fizzbuzz.java.c42b2a5c64ebaa24078385b2ae57284fe0ef501c36e276fe2461e7f8966e4fbf.jar' fizzbuzz
Error: Could not find or load main class fizzbuzz
Caused by: java.lang.ClassNotFoundException: fizzbuzz

8. Work around the previous issue by setting the JBANG_CACHE_DIR environment variable to something simple like C:\temp (which must exist)
9. Now jbang https://gist.github.com/maxandersen/f43b4c52dfcfc42dcd59a04e49acf6ec does work.
10. An example with external dependencies (from https://twitter.com/maxandersen/status/1466498299880476677): jbang --deps "org.openjfx:javafx-graphics:RELEASE:${os.detected.jfxname}" --java 17 https://raw.githubusercontent.com/bertjan/javafx-canvas-plasma/main/src/main/java/eu/openvalue/PlasmaApp.java - this, again, does not work:

[jbang] jbang version 0.83.1
[jbang] Retrieved file from cache https://raw.githubusercontent.com/bertjan/javafx-canvas-plasma/main/src/main/java/eu/openvalue/PlasmaApp.java = C:\dev\jbangcache\urls\5bf49b60633d87b02e4277671ab282ea998f5a22dd72350b7c1ea9d43d4a056e\PlasmaApp.java
[jbang] System Java version detected as 16
[jbang] System Java version 16 incompatible, using JBang managed version 17
[jbang] run: 'C:\dev\jbangcache\jdks\17\bin\java.exe' --module-path 'C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-base\18-ea+7\javafx-base-18-ea+7.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-graphics\18-ea+7\javafx-graphics-18-ea+7.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-base\18-ea+7\javafx-base-18-ea+7-win.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-graphics\18-ea+7\javafx-graphics-18-ea+7-win.jar' --add-modules javafx.base,javafx.graphics -classpath 'C:\dev\jbangcache\jars\PlasmaApp.java.a2ae658004be1bc8f88ae3d32a21288b25aa38642313976ef7036262444aefab.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-graphics\18-ea+7\javafx-graphics-18-ea+7.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-graphics\18-ea+7\javafx-graphics-18-ea+7-win.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-base\18-ea+7\javafx-base-18-ea+7.jar;C:\Users\Lßszl≤vandenHoek\.m2\repository\org\openjfx\javafx-base\18-ea+7\javafx-base-18-ea+7-win.jar' eu.openvalue.PlasmaApp
Error occurred during initialization of boot layer
java.lang.module.FindException: Module javafx.base not found

11. Set JBANG_REPO to something simple like C:\temp.
12. Clear the dependency cache because it's still pointing to the old Maven folder under the "special" home dir
13. re-attempt example with external dependencies; now it works.

Expected behavior The path is set correctly by the installer, and jbang works, without any manual fixes.

JBang version Not related to jbang version, only to installer.

Additional context After fixing the path manually, running jbang without parameters works. However, trying to do anything useful results in jbang resolving the "wrong" path again somehow. In other words, it is not possible to use jbang in a "native" shell with non-ASCII characters in your username. Changing the location of the cache and repo dirs fixes that, but that's not the out-of-the-box experience a new user would expect.

[quintesse]

That sounds like a code page problem, could you perhaps open a PowerShell window and execute this command:

[System.Text.Encoding]::Default.BodyName

And leave a comment with the result? Thanks!

[laszlovandenhoek]

The reported codepage is iso-8859-1.

I should add that I normally don't use PowerShell at all and that I have not (knowingly) altered any settings, so I would assume that this is the typical setup that native Windows 10 users have.

Also, in my defense, I did not pick this username myself! It was generated based on my identity in my organization's Office 365 environment. Over the years I've had PLENTY of experiences where my name turned out to be an untested corner case, and regrettably this one is no exception. Still, I figured I might make the best of it by reporting the issue (and describing a workaround, at least).

[laszlovandenhoek]

Regarding the issue: I'm struggling to find where in the installer script the Path environment variable is updated. Plenty of mentions of some "Path" or another, but all regarding physical files, it appears.

[laszlovandenhoek]

Update: running the simple FizzBuzz example from the manual doesn't work either. I have updated the issue description (step 7).

Interestingly, the Retrieved file from cache line still gets the path right, but then later the run: line points to the exact same wrong path I corrected earlier in the PATH environment variable.

[laszlovandenhoek]

I have updated the issue with further test cases. As a workaround, you need to set both JBANG_CACHE_DIR and JBANG_REPO environment variables to paths without "special" characters to get it to work.

[quintesse]

Thanks for all the work you're doing, it's really helpful :-)

So just to explain a bit: it's not the script that does it wrong, but the Java code. In this particular case the Java code prints out Powershell commands which then get executed by the script.

Somewhere along the line things get transformed from one encoding to another and things blow up. If you look at the --verbose output you'll actually see that Java knows perfectly well what the correct encoding should be and is able to print the path correctly. And then, just some lines below things are wrong.

Haven't figured it out yet, I'm debugging it now. (But it was a bit hard setting up a new dev environment for a new user heh)

[quintesse]

God do I hate Windows!

In CMD:

C:> chcp
Active code page: 437
C:> echo téstïng > x
C:> type x
téstïng

In Powershell (5.1 - default on my Windows):

C:> chcp
Active code page: 437
C:> echo téstïng > x
C:> type x
t‚st‹ng

In Powershell (7.1 - Newer version I installed):

C:> chcp
Active code page: 437
C:> echo téstïng > x
C:> type x
t�st�ng

WTF?

(edit: added another Powershell version)

[quintesse]

Jeez @maxandersen , not sure what to do with this. Right now I've tried every combination of setting Windows code pages and passing -Dfile.encoding=xxx to java and nothing works (lots of different result but none that makes a difference).

The only thing I've been able to make work so far is instead of printing the command to stdout and letting CMD or Powershell capture it, is to write it to a file and execute that. There must be some solution but so far it's eluding me :-/

[quintesse]

I've create a ZIP with a tiny test case that shows the problem without having to create a user. Just download, unpack and test. There's a README with some more instuctions.

issue1110.zip

[quintesse]

Just to show the output of that test ZIP in all cases:

CMD

C:\Temp\issue1110>.\testcmd.bat
Showing output from Java:
dir dir-téstïngShowing output from redirect:
dir dir-tΘst∩ng
Executing output from Java:
 Volume in drive C has no label.
 Volume Serial Number is 869D-441A

 Directory of C:\Temp\issue1110

File Not Found

NB: Weird thing is that the contents of the two files that are generated (one directly from the Java code (UTF8) and the other the redirected output (ANSI)) is completely fine. It's when reading the contents that stuff blows up.

Powershell 5.1

PS C:\temp\issue1110> .\testps.ps1
Showing output from Java:
dir dir-téstïng
Showing output captured in variable:
dir dir-tΘst∩ng
Showing output from redirect:
dir dir-tΘst∩ng
Executing output from Java:
dir : Cannot find path 'C:\temp\issue1110\dir-téstïng' because it does not exist.
At C:\temp\issue1110\out\outps1.ps1:1 char:1
+ dir dir-téstïng
+ ~~~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\temp\issue1110\dir-téstïng:String) [Get-ChildItem], ItemNotFoundEx
   ception
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

Executing output from redirect:
dir : Cannot find path 'C:\temp\issue1110\dir-tΘst∩ng' because it does not exist.
At C:\temp\issue1110\out\outps2.ps1:1 char:1
+ dir dir-tΘst∩ng
+ ~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (C:\temp\issue1110\dir-tΘst∩ng:String) [Get-ChildItem], ItemNotFoundExce
   ption
    + FullyQualifiedErrorId : PathNotFound,Microsoft.PowerShell.Commands.GetChildItemCommand

NB: In this case the contents of the file written by Java (UTF8) is fine, the other is not (UCS-2 LE BOM). Still, even though the Java version is okay it still messes up when reading

Powershell 7.1

PS C:\Temp\issue1110> .\testps.ps1
Showing output from Java:
dir dir-téstïng
Showing output captured in variable:
dir dir-tΘst∩ng
Showing output from redirect:
dir dir-tΘst∩ng
Executing output from Java:

    Directory: C:\Temp\issue1110\dir-téstïng

Mode                 LastWriteTime         Length Name
----                 -------------         ------ ----
-a---          03/12/2021    23:14              8 test1
-a---          03/12/2021    23:15              8 test2
-a---          03/12/2021    23:15              6 test3
Executing output from redirect:
Get-ChildItem: C:\Temp\issue1110\out\outps2.ps1:1
Line |
   1 |  dir dir-tΘst∩ng
     |  ~~~~~~~~~~~~~~~
     | Cannot find path 'C:\Temp\issue1110\dir-tΘst∩ng' because it does not exist.

NB: In this case the contents of the file written by Java is also fine, and the other is not (both are UTF8). So this is the only case where, without messing around with code pages and Java settings, we're able to get something that runs