Weird behavior of 'ivyDepsTree' in mill repo vs example projects on Windows

[malice00]

I am working on adding SBOM generation for mill projects into CycloneDX's cdxgen. Initially, I tested everything with the example projects, which worked just fine. When I wanted to test on a project that is a bit larger, I picked mill itself, seeing how it builds itself using itself. On MacOS and Linux, this worked perfectly well, but on Windows the generated trees are totally different from the ones generated in the examples!

In the examples and in mill on Mac/Linux, the trees look very nice, both in the console (maybe Windows no so much) and files (here Windows is fine). This is the 'bar' module from the java multi-module example:

├─ net.sourceforge.argparse4j:argparse4j:0.9.0
└─ org.thymeleaf:thymeleaf:3.1.1.RELEASE
   ├─ ognl:ognl:3.3.4
   │  └─ org.javassist:javassist:3.29.0-GA
   ├─ org.attoparser:attoparser:2.0.6.RELEASE
   ├─ org.slf4j:slf4j-api:2.0.5
   └─ org.unbescape:unbescape:1.1.6.RELEASE

Now, when trying to do this on mill itself, the characters used to build the tree are all replaced by '?', even in the files. This is 'bsp':

?? org.scala-lang:scala-library:2.13.15
?? com.lihaoyi:mill-moduledefs_2.13:0.11.2
   ?? com.lihaoyi:sourcecode_2.13:0.3.0
   ?  ?? org.scala-lang:scala-library:2.13.15
   ?? org.scala-lang:scala-compiler:2.13.15
   ?  ?? io.github.java-diff-utils:java-diff-utils:4.12
   ?  ?? net.java.dev.jna:jna:5.16.0
   ?  ?? org.jline:jline:3.28.0
   ?  ?? org.scala-lang:scala-library:2.13.15
   ?  ?? org.scala-lang:scala-reflect:2.13.15
   ?     ?? org.scala-lang:scala-library:2.13.15
   ?? org.scala-lang:scala-library:2.13.15

Is there a reason for this or is it actually some sort of bug? Of course I can build a workaround in my parser, but I'd much prefer if the behavior was the same with all versions of mill on all projects.

[lihaoyi]

@alexarchambault do you know if this is fixed by your recent work fixing a similar behavior in our integration/example test suite?

[lefou]

Looks like an encoding issue. @malice00 Which terminal do you use?

Is there a reason for this?

It looks like an character encoding issue to me. We either assume the wrong encoding or use a too limited encoding. The output itself comes from coursier library though.

[malice00]

I'm running this from Node with a default Windows setup (so that should be running in cmd.exe, if I remember correctly) and have tested with cmd.exe directly as well. I do however not think this is an encoding issue, seeing how I get different results depending on which project I generate the trees for... I can try with powershell tonight and report back, if you think that might help.

[lefou]

Can you mention the Mill versions you used to produce the nice charts (on all platforms). Could it be related to when Mill is run as a native launcher, since that it what Mill uses on itself in the main branch? (Look at the .config/mill-version)

[malice00]

I played around with the native launcher, but I admit I'm not sure what I used on the latest runs. I'll check and let you know when I get back home from work tonight.

[alexarchambault]

Trying things in a Windows VM, I confirm the tree doesn't print well from Mill (running mill -i main.ivyDepsTree from a Mill checkout, that uses Mill 0.13.0-M1-2-c0b707, prints a lot of ??). It does print well from the coursier CLI though (coursier resolve org.scala-lang:scala-compiler:2.13.16 -t prints a nice tree, that's using coursier 2.1.25-M4). I'm not sure why yet.

[malice00]

The example was using 0.12.7. I also tried it with 0.12.7-native, 0.12.10 and 0.12.10-native. Everything works as expected (just talking about the written files, the console is hot garbage every time).

Mill itself, was running 0.12.7-native (I was testing on tag 0.12.10). Tried the same as above and every single version just writes the question marks. I even tried the main branch, which has a 0.13 Milestone, but that branch doesn't even want to run for me -- it complains about files on a drive that I don't even have (D:)!

So, yeah, not sure why there is a difference between the example and mill, but it seems there is and for me it seems to be related to the project, not the version of mill being used.

[lihaoyi]

Probably the same as https://github.com/com-lihaoyi/mill/issues/6118

[lihaoyi]

Doesn't seem to be fixed, may need further windows+unicode fiddling https://stackoverflow.com/a/8921509/871202