`jupyter kernelspec list` returns wrong paths when run with Python (Microsoft Store)

[stephtr]

On my Windows 10 (1909) PC, I have installed Python 3.8 through the Microsoft Store. Due to Python running partially sandboxed, access to %APPDATA%\jupyter gets transparently redirected to %LOCALAPPDATA%\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\Roaming\jupyter. This is no issue while writing or reading files, but when running jupyter kernelspec list, the unredirected paths are outputted. That works fine if they are consumed by a (sandboxed) Python script, but not when an external tool (like the Python extension for VSCode) tries to use them.

After reading Python#40377, in my opinion it but be best if jupyter kernelspec list doesn't have to output any of its internal paths but would supply any necessary data through the cli.

In the meantime one can also create an environment variable JUPYTER_DATA_DIR pointing to C:\Users\<Username>\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.8_qbz5n2kfra8p0\LocalCache\Roaming\jupyter to evade the bug.

[MSeal]

Thanks for putting together such a detailed issue. There's a few different executions paths across repos that use the data directory listing to fetch files they care about. It would be on-trivial to change the interface to fetch data instead of paths. Would resolving the appdata redirect before returning the kernel list be a viable approach you think?

[stephtr]

Resolving the AppData redirection would in principle solve the issue (at least with today's Windows versions; I don't know what will happen in the future though). However, I didn't find an easy way to achieve that in native code; in Python it would be probably even harder to solve.

Another argument in favour of adding a new API over paths would be that other apps probably shouldn't mess around with jupyter's files but call the CLI instead. One wouldn't have to change every usage of paths, but could add a warning/deprecation message in the CLI that the outputted paths could be wrong under some circumstances.

[davidanthoff]

I'm currently trying to create a MSIX installer for Julia, and I've also run into a similar problem: any write operation to %APPDATA% from this MSIX installed Julia is virtualized and hidden from any other programs. The net effect is that when the IJulia package tries to install the Julia kernel, it succeeds, but the kernel won't be visible to any other program than julia.exe because in truth it has been written to a virtualized file location that is only visible to the MSIX installed julia.exe. So, none of the Jupyter clients will see the Julia kernel and nothings works...

I think the root problem here is that in the MSIX world, %APPDATA% is not a good location to share configuration between different programs. The philosophy behind MSIX (and Windows Store apps) seems to be that %APPDATA% is broadly speaking a per app store, and anything an app puts there is only visible to that app itself, and no other app.

I'm still investigating whether there is any workaround, but so far it is not looking promising. So I think the best way out here would be that Jupyter changes the file locations for kernels, I think the current %APPDATA%\jupyter\kernels just won't work very well in an MSIX/Windows Store world.

Maybe a simple way out would be to add another user location on Windows, say ~/.jupyter/kernels, where clients should look for kernels, in addition to all the existing locations? As far as I can tell, that location wouldn't be virtualized by MSIX.

Of course such an approach would require updates in all clients that currently have the list of folders hard coded, so it would take a while to role out...

[davidanthoff]

Here is the issue on the MSIX forum where I'm asking MS whether there is some workaround, for anyone interested in following that.

[MSeal]

Hmm thanks for digging into it some. Ironically, if the APPDATA env variable is not set it would actually use ~/.jupyter -- see jupyter_core.