Getting Started Repl doesnt start on WIndows w/ Oracle Java 15 and 16

[stuartstein777]

Issue Type: Bug

Click nREPL in bottom toolbar Click Fire Up Getting Started REPL Click Use Existing Directory repl fails to start with the error in the terminal:

The output window shows ; Jacking in... ; Starting Jack-in Terminal: java -jar ".calva\deps.clj.jar" -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.25.11""}}}" -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"

The terminal, on fail, shows: java -jar ".calva\deps.clj.jar" -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.25.11""}}}" -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]" Error while parsing option "--config-data {:deps {nrepl/nrepl {:mvn/version,0.8.3},cider/cider-nrepl {:mvn/version,0.25.11}}}": java.lang.NumberFormatException: Invalid number: 0.8.3 Jack-in process exited. Status: 1

Extension version: 2.0.194 VS Code version: Code 1.55.2 (3c4e3df9e89829dce27b7b5c24508306b151f30d, 2021-04-13T09:35:57.887Z) OS version: Windows_NT x64 10.0.18363

System Info

Item	Value
CPUs	Intel(R) Core(TM) i7-6820HQ CPU @ 2.70GHz (8 x 2712)
GPU Status	2d_canvas: enabled gpu_compositing: enabled multiple_raster_threads: enabled_on oop_rasterization: enabled opengl: enabled_on protected_video_decode: unavailable_off rasterization: enabled skia_renderer: enabled_on video_decode: enabled vulkan: disabled_off webgl: enabled webgl2: enabled
Load (avg)	undefined
Memory (System)	15.89GB (3.60GB free)
Process Argv	--crash-reporter-id 88f8d9b0-038d-493b-8525-6ef9c92ebe8c
Screen Reader	no
VM	0%

e to send. Please paste.

[bpringe]

Thanks for the report.

[PEZ]

Not sure I would label this a bug, at least not if the label is to indicate a bug in our code. It could be, but so far it seems to be rather upstreams and far upstream and tricky to figure out.

There are many layers to this. And some of those are unknown (at least to me), making it extremely difficult. For instance, it is unclear to me how the shell being used is determined. A while back I changed from using the Clojure CLI to deps.clj and a java command line. WorksOnMyMachine(s), and on many more of the ones where the Clojure CLI method didn't work.

(With Leiningen we use a cmd.exe command line.)

I am unsure what the ratio is between Windows machines where it is working and where it is not. Maybe it is 90/10, maybe something else... Everytime I try to fix this it seems to be improved (but what do I know), but also does not nail it 100%. The sample size is not big since most Windows users are wise enough to use WSL for development work.

Neither the Getting Started REPL nor the Standalone REPL can assume WSL. They need to work with the least common denominator. I might give up on this pipe dream at some point, and just tell Windows users that if the Getting Started REPL works, celebrate, and otherwise point them at some longer path to get to test the Clojure REPL plugged to the editor. But I am not there yet. (Probably should figure out how to inform about the longer path meanwhile, but anyway).

There are at least two parameters to the puzzle that I know of.

1. The quoting
2. A shell boolean I can pass to child_process.spawn()

The shell boolean determines if spawn() will use a shell to spawn the command line or just plain Windows. I am not very clear about what that means, but from what I have understood one difference is that more of the user's environment can be assumed to be present in the shell scenario. From experience I also know that it influences the effect of the quoting.

The current command line, let's denote it A, is built using a system where arguments are double quoted, let's denote DQ and double quotes in arguments use ”doubled double quotes”, let's call it DDQ and shell: true:

A: DQ+ DDQ + `shell:true``

• Command line:

  java -jar ".calva\deps.clj.jar" -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.25.11""}}}"  -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"
  

• WorksOnMyMachines(s)
• Maybe works on 90/100 machines?
• Doesn't work on @Stuart's nor @stianalmaas's machines
• The command line doesn't work from a PowerShell prompt
• The command line works from a CMD prompt

Despite not really having the time for this now I just conducted some experiments.

B: DQ + DDQ + shell:false

• Command line:

  java -jar .calva\deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.26.0""}}}"  -m nrepl.cmdline --middleware '[cider.nrepl/cider-middleware]'
  

• OnMyMachine fails in some similar ways regardless if from Calva or from prompts, PS or CMD.

I think this indicates that OnMyMachine PowerShell is used when shell:false. So trying with single quotes (SQ) and backslash quoted double quotes (BDQ).

C: SQ + BDQ + shell:false

• Command line:

  java -jar .calva\deps.clj.jar -Sdeps '{:deps {nrepl/nrepl {:mvn/version,\"0.8.3\"},cider/cider-nrepl {:mvn/version,\"0.26.0\"}}}'  -m nrepl.cmdline --middleware '[cider.nrepl/cider-middleware]'
  

• OnMyMachine Getting Started REPL command fails with

  Error building classpath. Don't know how to create ISeq from: clojure.lang.Symbol
  

• OnMyMachine this works from the PS prompt
• OnMyMachine this fails from the CMD prompt, with the exact same message as with the Getting Started REPL command

So, now maybe that indicates the command is being run through the CMD shell after all? At least OnMyMachine... What about other people's machines?

I'm stuck for now. Maybe I should try with a cmd.exe command line like with Leiningen. Even if I know I have tried that at some point too, but there are many moving parts so maybe the right combination has never been tried...

[borkdude]

You can diagnose how CLI args get passed through to Clojure command line args like this with babashka:

bb -e "*command-line-args*" -- 1 2 3
("1" "2" "3")

E.g. in cmd.exe:

bb -e "*command-line-args*" -- -Sdeps "{:mvn/version """1.10.3"""}"
("-Sdeps" "{:mvn/version \"1.10.3\"}")

In Powershell:

bb -e "*command-line-args*" -- -Sdeps '{:mvn/version \"1.10.3\"}'
("-Sdeps" "{:mvn/version \"1.10.3\"}")

I don't understand what cmd.exe and Powershell have to do with this if you are launching the process from NodeJS. The shell isn't involved in that case. When shelling out from Java on Windows I usually have to apply this trick:

https://github.com/babashka/process/blob/eba04e6c35814296e5bb383804f801aee912cba1/src/babashka/process.clj#L152

PS: you can install babashka on Windows easily with scoop:

https://github.com/littleli/scoop-clojure

[PEZ]

Thanks!

I don't understand what cmd.exe and Powershell have to do with this if you are launching the process from NodeJS. The shell isn't involved in that case.

I don't really understand this, as must be obvious from my comment above. But something is interpreting the command line and needs quoting of some kind. And it seems that this something is different on different machines. Also I want the command line echoed to work if the user copies it and pastes it on the prompt. In which case it is good to know which kind of prompt. And, since the shell option to spawn() makes a difference in the quoting, I think maybe the shell is involved in that case, even if I don't know which shell.

I'll try that trick of yours now...

[borkdude]

From https://nodejs.org/api/child_process.html#child_process_spawning_bat_and_cmd_files_on_windows:

Spawning .bat and .cmd files on Windows# The importance of the distinction between child_process.exec() and child_process.execFile() can vary based on platform. On Unix-type operating systems (Unix, Linux, macOS) child_process.execFile() can be more efficient because it does not spawn a shell by default. On Windows, however, .bat and .cmd files are not executable on their own without a terminal, and therefore cannot be launched using child_process.execFile(). When running on Windows, .bat and .cmd files can be invoked using child_process.spawn() with the shell option set, with child_process.exec(), or by spawning cmd.exe and passing the .bat or .cmd file as an argument (which is what the shell option and child_process.exec() do). In any case, if the script filename contains spaces it needs to be quoted.

Since you are launching java I think you can get away with NOT using the shell option (unlike when you are running the clojure CLI which is a shell script).

[borkdude]

FWIW, this older clj-kondo plugin for VSCode ran the clj-kondo binary in this way:

https://github.com/marcomorain/clojure-lint/blob/86f699258d7478d44237241ecf1f0be914170887/src/extension.ts#L55

AFAIK it worked cross-platform (and probably still works)

[PEZ]

I don't think we can use execFile() because Calva jack-in is not fire-and-forget, we need to be able to communicate with the process. Anyway, we use spawn() in about an identical way as in that example, which never enters double quote hell.

[PEZ]

Since you are launching java I think you can get away with NOT using the shell option (unlike when you are running the clojure CLI which is a shell script)

I think so too. Just haven't succeeded with that yet. Not for lack of trying.

[lread]

@PEZ at one point I decided I would figure Windows shell escaping out and started taking notes.
I did not get very far before my mind got very numb. Here are my nascent notes for what they are worth: https://github.com/lread/info-process-builder

[PEZ]

Thanks @lread . I'll certainly check that out because I have also tried to lay this puzzle and I get dizzy. 😄

[PEZ]

The quoting problems seems to have been a red herring. In my Windows dev environment I have removed the quoting out of the picture (latest #1164 VSIX also has this build) which worked on that machine until I upgraded Java.

So the failure seems to be introduced somewhere between Java 8 and Java 15, I haven't bisected it down further than that right now.

Now I am actually stuck again. Can't think of a single thing I can try to make this work. I am hoping som sleep will help.

[PEZ]

I have now tried with Java 12 and the command works then. So the error is then introduced with Java 13, 14 or 15, I guess. Here is some sort of reproduction:

From a directory with deps.clj.jar in it (the .calva directory in the temp folder created by the Getting Started REPL will do):

1. Start a CMD prompt (if you are using a VS Code terminal use the dropdowns or execute cmd from the PS prompt)
2. Execute the following command:

   java -jar deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.26.0""}}}"  -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"
   

Expected: The nREPL server is started

Actual, using Java 12: The nREPL server started

Actual, using Java 15 and 16:

Error: Could not find or load main class clojure.main
Caused by: java.lang.ClassNotFoundException: clojure.main

Now I would want to try this with the Clojure CLI as well, but I can't get that to work on my Windows machine. So if someone with the Clojure CLI working on Windows could try this that would be awesome. (Just replace java -jar deps.clj.jar with clojure on the above command line.)

NB: Not that Clojure CLI is an option for the Getting Started REPL, which only requires Java. It is for trying to locate where the error happens.

[PEZ]

Removed some confusing comments. It seems that even if java -version reports a version, it is not necessarily what is used when actually running the command. Creating a new shell after installing java15 actually reproduces the error.

[PEZ]

And now I think I have confirmed that the error seems to happen only with the Java 15 and 16 (and possibly earlier versions) installed with Oracle's installer. Using adopt15-hotspot installed with scoop the Getting Started REPL works with currently released Calva as well as with the this #1164 VSIX:

• https://13280-125431277-gh.circle-artifacts.com/0/tmp/artifacts/calva-2.0.195-1162-getting-started-repl-win-076d5d2f.vsix

Which does not add any quoting what so ever to the command shelled out on Windows.

[stianalmaas]

I have tried to test some variations on my machine, since it looks like @stuartstein777 have similar issues. If I run from Windows cmd it works, on all versions of java that I have.

Here with Java 16: `C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk.calva>java -version java version "16" 2021-03-16 Java(TM) SE Runtime Environment (build 16+36-2231) Java HotSpot(TM) 64-Bit Server VM (build 16+36-2231, mixed mode, sharing)

C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk.calva>java -jar deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.26.0""}}}" -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]" nREPL server started on port 54590 on host kubernetes.docker.internal - nrepl://kubernetes.docker.internal:54590`

Here with Java 11: `C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk.calva>java -version java version "11.0.11" 2021-04-20 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.11+9-LTS-194) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.11+9-LTS-194, mixed mode)

C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk.calva>java -jar deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.26.0""}}}" -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]" nREPL server started on port 54775 on host kubernetes.docker.internal - nrepl://kubernetes.docker.internal:54775`

And now the Getting started REPL in Calva also works!

This one is a proper Heisenbug indeed.

What I did was open a powershell in VS Code. There I started the cmd shell and ran the command : java -jar deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,""0.8.3""},cider/cider-nrepl {:mvn/version,""0.26.0""}}}" -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]" I had Java 11 active. This downloaded and started up the nREPL server.

After that every combination I have tried has worked. I have set Java 16 in JAVA_HOME and path, and started from cmd. And now it also works in Calva.

java -jar .calva\deps.clj.jar -Sdeps {:deps {nrepl/nrepl {:mvn/version,"0.8.1"},cider/cider-nrepl {:mvn/version,"0.26.0"}}} -m nrepl.cmdline --middleware [cider.nrepl/cider-middleware] Downloading: nrepl/nrepl/0.8.1/nrepl-0.8.1.pom from clojars Downloading: nrepl/nrepl/0.8.1/nrepl-0.8.1.jar from clojars nREPL server started on port 55059 on host kubernetes.docker.internal - nrepl://kubernetes.docker.internal:55059

Here I have downgraded the nrepl version to see if the download affects something. It didn't.

[PEZ]

Thanks, @stianalmaas !

How does docker and kubernetes fit into your setup? Can you also paste the output from the Jack-in terminal, when it works?

[stianalmaas]

I have Docker Desktop for Windows installed. Isn't it normal that the nREPL server uses that?

Here is what the output looks like when starting up:

; Jacking in...
; nREPL Connection was closed
; Starting Jack-in Terminal: java -jar .calva\deps.clj.jar -Sdeps {:deps {nrepl/nrepl {:mvn/version,"0.8.1"},cider/cider-nrepl {:mvn/version,"0.26.0"}}}  -m nrepl.cmdline --middleware [cider.nrepl/cider-middleware]
; Hooking up nREPL sessions...
clj꞉user꞉> 
; Jack-in done.
clj꞉user꞉> 
; Evaluating file: hello_repl.clj
"hello_repl.clj is loaded, and ready with some things for you to try."
clj꞉hello-repl꞉> 

[lread]

@PEZ, I haven't been following your attempts and strategies closely, and maybe you have thought of this already but:

I wonder if deps.clj could be modified to retrieve its args from a specified generated file rather than the command line. This way you could remove Windows escaping rules from the equation.

So maybe you'd have:

java -jar .calva/deps.clj.jar -Sargs-file some-generated-args-file.edn

Would something like that maybe help?

[stianalmaas]

@PEZ I also managed to get it working with the Clojure CLI now. For that I have to use Powershell. The trick here was to escape the double-quotes with back-ticks. Like this:

PS C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk\.calva> clojure -Sdeps "{:deps {nrepl/nrepl {:mvn/version,`"`"0.8.3`"`"},cider/cider-nrepl {:mvn/version,`"`"0.26.0`"`"}}}"  -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"
WARNING: When invoking clojure.main, use -M
nREPL server started on port 58254 on host kubernetes.docker.internal - nrepl://kubernetes.docker.internal:58254

This also works with Java, so it seems everything works no matter what I do...

PS C:\Users\stian\AppData\Local\Temp\betterthantomorrow.calva\ur5dk\.calva> java -jar deps.clj.jar -Sdeps "{:deps {nrepl/nrepl {:mvn/version,`"`"0.8.3`"`"},cider/cider-nrepl {:mvn/version,`"`"0.26.0`"`"}}}"  -m nrepl.cmdline --middleware "[cider.nrepl/cider-middleware]"
nREPL server started on port 58273 on host kubernetes.docker.internal - nrepl://kubernetes.docker.internal:58273

But maybe it is better to think outside the box as @lread suggests.

[PEZ]

Latest findings is that

1. Clojure CLI suffers from the same problems
2. It so far only is related to the java executable that the Oracle installer wires up as the current

Using the command line from https://clojure.atlassian.net/browse/TDEPS-133#icft=TDEPS-133 as an example:

PS C:\Program Files\Common Files\Oracle\Java\javapath> .\java.exe -jar .\deps.clj.jar -Sdeps '{:deps {viebel/klipse-repl {:mvn/version \"0.2.3\"}}}' -m klipse-repl.main
Error while parsing option "--config-data {:deps {viebel/klipse-repl {:mvn/version 0.2.3}}}": java.lang.NumberFormatException: Invalid number: 0.2.3
PS C:\Program Files\Common Files\Oracle\Java\javapath>

But:

PS C:\Program Files\Java\jdk-16.0.1\bin> .\java.exe -jar .\deps.clj.jar -Sdeps '{:deps {viebel/klipse-repl {:mvn/version \"0.2.3\"}}}' -m klipse-repl.main
Welcome to Klipse REPL (Read-Eval-Print Loop)
Clojure 1.10.3
user=>

Both experiments need to be run with this setting of JAVA_CMD:

PS C:\ ...\bin> $env:JAVA_CMD = ".\java.exe"

(No, it doesn't matter if we try with other quoting. It is not related to our quoting.)

Note that the Oracle installer installs both these executables. One is fine, the other is not. Problem is that it makes the one not working the default, and that cause the Getting Started REPL to fail.

[bpringe]

Great isolation of the problem! This has been quite a journey that I've been following.

[PEZ]

@lread, I doubt that would help. The args are delivered just fine to java and to deps.clj.

[borkdude]

@PEZ Can you also add to the details whether you set the environment variable JAVA_CMD to the local directory's java.exe executable? This is important to mention because if you don't set it, then deps.clj will pick up the java.exe that is on the system's PATH.

[borkdude]

To check from a REPL which Java version you are running: (System/getProperty "java.version") and (System/getProperty "java.vendor").

[PEZ]

Updated the comment above with info about setting JAVA_CMD.

[PEZ]

Further narrowing it down:

PS C:\Users\Peter Strömberg> java -jar deps.clj.jar -M -r -Sdeps '{:deps {viebel/klipse-repl {:mvn/version \"0.2.3\"}}}'
Clojure 1.10.3
user=> *command-line-args*
("-Sdeps" "{:deps {viebel/klipse-repl {:mvn/version 0.2.3}}}")
user=>
PS C:\Users\Peter Strömberg> cd 'C:\Program Files\Java\jdk-16.0.1\bin\'
PS C:\Program Files\Java\jdk-16.0.1\bin> $env:JAVA_CMD = ".\java.exe"
PS C:\Program Files\Java\jdk-16.0.1\bin> .\java.exe -jar deps.clj.jar -M -r -Sdeps '{:deps {viebel/klipse-repl {:mvn/version \"0.2.3\"}}}'
Clojure 1.10.3
user=> *command-line-args*
("-Sdeps" "{:deps {viebel/klipse-repl {:mvn/version \"0.2.3\"}}}")
user=>

[PEZ]

Some further narrowing, with very active coaching from @borkdude. This program:

public class PrintArgs {
    public static void main(String [] args) {
        for (String arg: args) {
            System.out.println(arg);
        }
    }
}

With system wide Java:

PS C:\Program Files\Java\jdk-16.0.1\bin> java PrintArgs '\"bar\"'
bar

With the java.exe in C:\Program Files\Java\jdk-16.0.1\bin (installed by the same installer):

PS C:\Program Files\Java\jdk-16.0.1\bin> .\java.exe PrintArgs '\"bar\"'
"bar"

I've now filed a bug report about it, because I failed to find it in the database: https://bugs.java.com/bugdatabase/

[borkdude]

Note that the error (losing the quotes in the first example) above happens with Oracle JDK 16 as the "system" Java and does not occur with any other known Java installations like AdoptOpenJDK or with any lower Oracle versions (like 11). Important to include these details for context...

[borkdude]

Also note that @PEZ could not pass Java properties with the "system" Oracle JDK 16:

PS C:\Program Files\Java\jdk-16.0.1\bin> java -Djdk.lang.process.allowAmbiguousCommands=true PrintArgs '\"bar\"'
Error: Could not find or load main class .lang.process.allowAmbiguousCommands=true

I'm not sure what we're dealing with here, it looks like the Oracle JDK 16 wrapper is seriously screwing up command line parsing in multiple ways. Have you included this in the issue Oracle issue @PEZ? If not, I think it warrants another issue.

[eerohele]

This might've been covered by one of the earlier posts in this thread, but this workaround (for a project REPL, not for a getting started REPL — sorry!) works on my colleague's Windows machine:

That is, wrap every value in the Value column in double double quotes.

Java version:

java version "15.0.2" 2021-01-19
Java(TM) SE Runtime Environment (build 15.0.2+7-27)
Java HotSpot(TM) 64-Bit Server VM (build 15.0.2+7-27, mixed mode, sharing)

Output of where java:

C:\Program Files\Common Files\Oracle\Java\javapath\java.exe
C:\Program Files (x86)\Common Files\Oracle\Java\javapath\java.exe