Deal with Windows support being an ongoing shit show

[soc]

Many contributors have spent heroic efforts to keep Windows support working, for which I'm greatly thankful.

Though it appears to me that Windows support keeps breaking time and time again – perhaps it's time to think whether a different approach could be less painful and more respectful to the time & efforts of contributors?

I'm short on actual ideas though:

• Alternative approaches that DON'T work:
  • Reading environment variables.
  • Reading registry values.
  • Shipping pre-compiled native code.
• Alternative approaches that may work:
  • Shipping a .NET assembly that replaces the current Powershell + .NET part?

I'm open to suggestions, thoughts, ideas, etc. – what do people (@alexarchambault, @eatkins, @fthomas. @phongngtuan, ...) think?

[phongngtuan]

pardon my ignorance but what is the issue with System.getenv("APPDATA") and System.getenv("LOCALAPPDATA") ?

[alexarchambault]

I think embedding a tiny JNI library could work too, as we're only calling simple system calls. I had started toying with that some time ago. IIRC, the main hurdle I ran into was charset conversions, from what SHGetKnownFolderPath gives us, to something JNI accepts as a string.

[alexarchambault]

@phongngtuan It's discussed below https://github.com/dirs-dev/directories-jvm/issues/26#issuecomment-570288952, these environment variables are not always in sync with the system call, which returns the true values.

[alexarchambault]

About the charset issue I mentioned above, maybe wcstombs and NewString would just work...

[eatkins]

I feel strongly that precompiled native code with fallback to shelling out is the way to go but I also understand why @soc has skepticism. My experience is that while JNI is definitely a pain and does come with risks, it can be incorporated successfully. For sbt, I wrote a library that speeds up recursive directory listing by bundling pre-compiled code for some platforms. If the library is unable to load the native code for any reason it falls back to jvm built-ins. This code shipped with sbt 1.3.0 and there has only been one issue that came up in pre-release with the windows code: https://github.com/sbt/sbt/issues/4690. Perhaps unsurprisingly given @alexarchambault's comment above, it was also related to Window's use of wide characters by default: https://github.com/swoval/swoval/commit/e425e4c867ce24247da6a53365073849aa44d8d7 and was basically resolved by switching from NewStringUTF to NewString.

Loading a small dll and running it is generally much faster than shelling out to a process. There are certainly safety issues with using c but I think for this specific use case they are minimal because naively it seems as though the JNI code would just make a single system call and return a java value back to the jvm. This means that it wouldn't need to use the heap, mitigating one of the biggest worries with native code.

I am not personally interested in working on this but I would be happy to share my advice and review any changes. I have spent a lot of time incorporating native code into jvm libraries. In addition to swoval, I also added jni support for unix domain sockets and windows named pipes so that we could use them in a graalvm native image for the sbt thin client (JNA uses reflection in a way that the graalvm could not handle): https://github.com/sbt/ipcsocket/pull/8. There are a few avoidable gotchas if the project decides to go down that road.

[alexarchambault]

So I have this working. I also manually checked that it works with non-ASCII characters.

I think I'm going to try to have it built and published from GitHub actions, and have ProjectDirectories.fromPath accept a custom getWinDirs method as input, so that I can have it use this library, and see how it works.

[alexarchambault]

For context, I think the JNI / Powershell approach is slightly better than just reading env vars, but more importantly, it should be useful in other places in coursier (terminal-related stuff, Windows env var stuff), which is why I'm trying to stick to it.

[soc]

@eatkins My main concern is that I'd really like to avoid having yet-another layer on top of the already existing layers that can fail (or not apply due to obscure reasons/on obscure platforms). I'd like to have something that reduces the existing complexity while increasing the reliability.

I'll be migrating this library to the new Java FFI as soon as Project Panama ships, but in the meantime I think a stop-gap solution that is less brittle than what we currently have is direly needed.

[matejsarlija]

https://i.imgflip.com/5refek.jpg (had to do this, this issue pops up on every Scala tool).

[soc]

@RayKoopa welcome!

[RayKoopa]

Hey there, crossposting my thoughts on this. The scenario here seems to be to remove the PowerShell invocation as it is causing trouble. These options seem viable, which you've mentioned in part above:

• Push the SHGetKnownFolderPath call from your PS script into a .NET assembly and call the assembly instead.
• Use JNI / C directly to do the SHGetKnownFolderPath call in there.
• Find a PowerShell expert knowing how to solve the PS invocation issues, if it is generally possible at all (I'm afraid I'm not one).

I don't know security implications of JNI as I'm not a Java developer, but since you effectively call a C WinAPI method, the call itself wouldn't be more or less secure than doing it in .NET.

If you have considered using .NET's base library method System.Environment.GetFolderPath, I have to disappoint you:

• It internally just calls SHGetKnownFolderPath too, but its signature is limited to the enum values defined in System.Environment.SpecialFolders.
• Given this, it does not not support the "Downloads" and "Public" folder as the enum is based on folders that existed in Windows XP only (the enum internally maps to values intended for a deprecated WinAPI call which never supported these folders, despite .NET now internally calling the new Windows Vista+ API, but not having extended that enum due to those old values).
• Support for the "new" (15+ years 😎) folders introduced in Vista is now mainly discussed in https://github.com/dotnet/runtime/issues/554 and may ship in .NET 7 or 8. However, these .NET versions are not preinstalled by Windows (Update) and require to be installed manually. Additionally, if it doesn't make the cut for .NET 7 and only .NET 8, Windows 7 may not be supported as its extended support period ends before .NET 8's scheduled release (end of 2023). Also Windows Vista is not supported by the new .NET (Core) at all.

Given this, I'd personally recommend just using JNI / C for this. It seems to be the natural thing Java does for native OS-specific API calls anyway, I presume?

If you still want to P/Invoke in .NET, you may like my CodeProject article on that. As mentioned in the dotnet issue, your call in the PS script seems effectively correct, though you can / should use automatic string marshaling to not have to deal with manually converting the returned buffer to a .NET string, and freeing it in all cases afterwards.

Also, a side note on the "Public" folder: On Windows, this folder is per-system, not per-user. It also has subfolders for Documents, Downloads, Pictures, etc., but these subfolders can be redirected, even outside of the parent "Public" folder. If a user wants to store a "public document" and expects it to be in Public > Documents, they would have to explicitly query the "PublicDocuments" path, since appending "\Documents" to the public path is out of the question due to the redirection scenario.

[soc]

@RayKoopa Thank you! My concern with JNI/C is that I have to maintain/keep/compile a piece of code for every platform Windows runs on. I own a Windows license for exactly zero platforms, and cross-compiling with C seems to be painful as well.

Which brings us to using C (compilers), which I'd rather just not. "It can be written safely" is probably true for this use-case, but too many people making this verdict in too many situations is exactly what brought computing into disrepute.

I'm starting to wonder whether the registry values are generally reliable. So Fonts may not exist, but on the other hand even Downloads is in there. I wonder ...

• ... what the registry keys look like on these Windows CI instances and other minimal installs
• ... whether they are updated to reflect changes after calls to SHSetKnownFolderPath

[RayKoopa]

Hmm I see. I hoped there would be some kind of interoperability offered by JNI that would not require to compile raw C and introduce lots of complexity to your project; kinda like P/Invoke in .NET. EDIT: I just found out about JNA, example usage. This looks like somebody already wrapped it for you quite nicely. /EDIT

Maybe you can find another way; effectively you "just" need to do two standard C calls made by something preinstalled playing together better with Java, and convert the returned memory buffer storing a 0-terminated UTF16 LE string to a Java String. 🤔

As you already know, the registry method is unsupported. ~~In practice, to my experience, at very weird occassions the keys do not exist or can be outdated / wrong. Required reading would be https://devblogs.microsoft.com/oldnewthing/20031103-00/?p=41973 and https://devblogs.microsoft.com/oldnewthing/20110322-00/?p=11163 . It may however be "better" to risk "possibly" incorrect results rather than getting no results due to PS invocation failing (which I know nothing about sadly). Your mileage may vary.~~ I just checked the registry key on my system: It does not list the "Public" folder you are seeking support for, nor for the subfolders included in it. So the registry solution is not only unsupported, it is also insufficient.

[eatkins]

@soc your concerns about maintainability are quite well warranted and I am sympathetic. I can't remember if this came up on other threads but is there a reason why JNA is not considered an option here? I personally kind of hate JNA and would rather just write a JNI library but I think it may alleviate the cross compilation problem. Adding a dependency would be unfortunate but sbt, which I'm guessing is the biggest consumer of this library, already depends on the JNA anyway.

Either way, I will reiterate that I don't think the JNI solution is as bad as it may seen. Microsoft for all its many flaws religiously preserves backward compatibility so you just would need a system for compiling the binary once and then checking it in (though I will acknowledge that checking binaries in to a git repo is icky). I have had success cross-compiling jni libraries with mingw, which is available for mac and linux, for x86_64 though I have never tried it for an arm platform. It also would be possible to compile on a CI platform like appveyor (I have built and distributed binaries using appveyor) or github actions (I haven't actually tried this but would assume that github actions would have a mechanism for exporting artifacts). It would indeed be unreasonable for you to be expected to maintain code for platforms that you cannot easily test so you could reasonably draw a line in the sand that any supported windows platform must have freely available cloud vms to build any needed binaries.

I am very sympathetic to the desire to avoid JNI or any windows specific building. I don't have a windows license either. The way I handled it was to install windows on virtualbox, which does not require a paid license. It was never pleasant but it worked. I also agree with you ideologically that c is terrible but if the only api the OS provides is a c api, I'm not sure what option there is but to submit to working directly with that api.

[eatkins]

I am going to unsubscribe from this thread. Every time there is a comment, it is like a mini ddos attack on my brain. It is frustrating that from my perspective the only viable solution is being rejected for ideological reasons. This is a disservice to the downstream users of the library who are affected. If someone needs help implementing a jni solution, I have experience and would be happy to offer assistance. Please email me directly and don’t @ me here.

[soc]

is there a reason why JNA is not considered an option here

Sadly JNA is rather big, it would turn this < 10kB library into a 3.7MB one. :-/ (see https://github.com/dirs-dev/directories-jvm/issues/16)

[alexarchambault]

If ever people stumble upon this issue, this is addressed in coursier, via JNI, by:

• using coursier/directories-jvm, that has https://github.com/coursier/directories-jvm/commit/7acadf2ab9a4ce306d840d652cdb77fade11b94b
• using coursier/jni-utils to create a GetWinDirs instance to be passed as fourth parameter to ProjectDirectories.from, like done here

coursier/directories-jvm isn't published on Maven Central. coursier uses it as a source dependency, then shades it. So if you depend on coursier, you can access it via coursier.cache.shaded.dirs.ProjectDirectories, and get the from method accepting a fourth parameter.

I didn't test it on Windows ARM64, so I have no idea how it works there (but coursier has a fallback to former powershell stuff in that case).

[soc]

@alexarchambault good work! Do you know how large the binaries turned out?

I'm experimenting with an approach in dirs-cli and end up with 30KiB (16KiB with upx). Though I still need to replace the Unix-only File::from_raw_fd(1) with WriteConsole for Windows.

Another thought is building an x86 binary and using that on both x86-64 and x86. (Not sure what's the situation on ARM...)

[DavidGregory084]

@soc If it helps, you can implement JNI functions in Rust without too much trouble (see e.g. this code here) so you could call through to your directories-sys-rs crate rather than implementing something in C, and it would then support the same platforms as your Rust code does? AFAIK the approach that folks usually use to deploy these is to bundle the dylib for every platform into the resources of the JAR file, then extract the one for the current platform into a temp directory and load it using System.load at static initialization time, although I have never implemented anything like this myself. I can't think of a library example off the top of my head but protoc-jar does something similar.

[soc]

@DavidGregory084 Interesting, thanks!

[DavidGregory084]

@soc this library might help with the loading native libs part.

I found that ZeroMQ uses this library to load a bunch of libraries in its JNI bindings: https://github.com/zeromq/czmq/tree/master/bindings/jni/czmq-jni.

That project uses gradle to create the final JAR file in the structure expected by the native-lib-loader.

[soc]

@soc this library might help with the loading native libs part.

Thanks @DavidGregory084, I'll have a look!

[soc]

I have some experimental code using the new FFI API of Java 22.

That sadly doesn't help all those you are going to be stuck on Java < 22 for the next few years, but perhaps it can serve as a "known good" implementation.

[brcolow]

I posted this in another PR but this may be more a appropriate place:

I made a proof-of-concept using Java 22 Foreign Function & Memory API of how to extract a LocalAppData (as an example) known folder id, in case it is helpful for you.

https://gist.github.com/brcolow/e6c2e59a3aa29d32d3332bcf10313031

[soc]

Hi @brcolow, thanks for the code! That looks way more fleshed out than my efforts. I'll give it a go next week.