julia
julia copied to clipboard
JuliaLang
Permissions issues with artifacts on Windows in 1.10-rc1 (and betas) - Access is denied
This is a somewhat speculative issue without a reproducer - I'm filing this as I've run into this a few times now and have had sporadic interactions with people on Discourse/Slack seeing this as well, so hopefully this consolidates things by giving those who experience this issue something to search for.
What prompted me to file today was seeing this in one of my environments:
64 dependencies successfully precompiled in 360 seconds. 266 already precompiled.
1 dependency had output during precompilation:
┌ GR
│ ERROR: LoadError: InitError: could not load library "C:\Users\ngudat\.julia\artifacts\52bbefbea6a9098fa5c57208812d3868fac90841\bin\libxml2-2.dll"
│ Access is denied.
│ Stacktrace:
│ [1] dlopen(s::String, flags::UInt32; throw_error::Bool)
│ @ Base.Libc.Libdl .\libdl.jl:117
I had previously posted on Discourse about this here:
https://discourse.julialang.org/t/windows-artifact-issue-access-denied-when-installing-plots-makie/103734/4
with others chiming in reporting the same issue. There's also an issue reported with Yggdrasil here:
https://github.com/JuliaPackaging/Yggdrasil/issues/7625
All of this is a bit of a throwback to this old issue from 1.5 days:
https://github.com/JuliaLang/julia/issues/38411
(and again with what I've seen on 1.10 again starting Julia as admin was fine, but then required to always run as admin when wanting to use any packages loading artifacts).
If anything changed in how artifacts get handled on Windows it might be worth looking into. Equally if no one else chimes in here reporting any issues I might just be seeing a hangover effect from artifacts installed earlier and the problem is solved on rc1 already.
This seems to have become more widespread?
https://discourse.julialang.org/t/access-denied-for-plotting-artifacts/110570/2
https://discourse.julialang.org/t/given-up-installing-any-plotting-packages/110272/24
I believe this is the corresponding Pkg issue https://github.com/JuliaLang/Pkg.jl/issues/3269
I believe this is the corresponding Pkg issue https://github.com/JuliaLang/Pkg.jl/issues/3269
I'm not sure, the error reported here has been observed only on Windows and only on Julia v1.10+, the Pkg issue you linked is on Linux and it started already in Julia v1.8.
I feel like this may be related (again) to https://github.com/JuliaLang/Pkg.jl/pull/3349
Is there anything I can do to help with this? I'm currently not affected by the issue but - at risk of sounding annoying - this is the sort of issue that really puts off the typical corporate engineering/data science user that is usually on Windows and checking out Julia, only to find that they can't use any package that has an artifact somewhere in its dependency chain (which is almost surely at least one package for a new user).
I appreciate that the Venn diagram between "can contribute to Julia internals" and "uses Windows" is pretty much empty, but I'm happy to try out stuff if there is someone in the first part of the Venn diagram who has ideas about how this could be tackled.
In the linked issue, @staticfloat wrote that this was happening because Pkg is installing things read-only. Do dlls need to be writeable on Windows or something?
Do dlls need to be writeable on Windows
I don't believe that's the case. Without some kind of reproducer it's going to be very difficult to track down what's happening here.
Do any of these reproduce? They seem to have specific instances of failing packages.
https://github.com/JuliaPackaging/Yggdrasil/issues/7625 https://github.com/JuliaLang/julia/issues/53139
EDIT: hiding this comment since there is no difference between the artifacts, because I copied the artifact dir over without NTFS permissions, leading in the wrong direction.
I have no reproducer because they happen rarely, but I did post the stat of such a DLL in https://github.com/JuliaPackaging/Yggdrasil/pull/7412#issuecomment-1731613331. When I posted that stat I thought the read-only mode was the problem because if I removed the artifact and reinstalled it somehow wasn't read-only anymore. But now with 1.10.1 I see all my artifacts are read-only and they usually work fine.
I did save the bad artifact dir of the [email protected] in question at the time. I put a good version I just downloaded next to it and uploaded it temporarily here. Can anyone spot any differences? To me it looks like the stat.mode and binary contents are identical, but if I put the bad artifact in .julia/artifacts I always get this on using ICU_jll:
ERROR: InitError: could not load library "C:\Users\visser_mn\.julia\artifacts\2a29863b092214f3e985df7ccf7601ae41d8f406\bin\icuin69.dll"
Access is denied.
Stacktrace:
[1] dlopen(s::String, flags::UInt32; throw_error::Bool)
@ Base.Libc.Libdl .\libdl.jl:117
And if I put the good one, everything works fine.
Not necessarily a reproducer, as this problem does not seem to appear in all windows machines. But on my machine, there is an issue always encountered with PlotlyKaleido since using julia 1.10 that depends on the artifact folder of Kaleido_jll being read-only.
This means that on windows I can't even cd into that folder:
julia> using Kaleido_jll
julia> cd(Kaleido_jll.artifact_dir)
ERROR: IOError: cd("C:\\Users\\Alberto.Mengali\\.julia\\artifacts\\7914a56da888d6a06d00c87f97e873c60e97acc7"): permission denied (EACCES)
Stacktrace:
[1] uv_error
@ .\libuv.jl:100 [inlined]
[2] cd(dir::String)
@ Base.Filesystem .\file.jl:91
[3] top-level scope
@ REPL[4]:1
This is likely what causes the problems with running the kaleido library as on windows this relies on calling a script called kaleido.cmd located in the artifact folder.
The contents of this script are quite simple:
@echo off
setlocal
chdir /d "%~dp0"
.\bin\kaleido.exe %*
And when trying to execute that, the following error messages are thrown:
julia> let path = joinpath(Kaleido_jll.artifact_dir, "kaleido.cmd")
run(`$path`)
end
Access is denied.
The system cannot find the path specified.
ERROR: failed process: Process(`'C:\Users\Alberto.Mengali\.julia\artifacts\7914a56da888d6a06d00c87f97e873c60e97acc7\kaleido.cmd'`, ProcessExited(1)) [1]
Stacktrace:
[1] pipeline_error
@ .\process.jl:565 [inlined]
[2] run(::Cmd; wait::Bool)
@ Base .\process.jl:480
[3] run(::Cmd)
@ Base .\process.jl:477
[4] top-level scope
@ REPL[5]:2
The first error Access is denied is triggerd by the chdir /d "%~dp0" line in the script above, which fails to change directory to the artifacts dir, and after failing that, the execution of .\bin\kaleido.exe %* failes to find the executable as the path is relative and the folder is wrong.
Edit:
Adding output of versioninfo()
julia> versioninfo()
Julia Version 1.10.1
Commit 7790d6f064 (2024-02-13 20:41 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 20 × 12th Gen Intel(R) Core(TM) i7-12800H
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, alderlake)
Threads: 20 default, 0 interactive, 10 GC (on 20 virtual cores)
Environment:
JULIA_PKG_USE_CLI_GIT = true
Maybe it isn't executable? If that's the case and also the library isn't executable, that'd explain the issue
I also cannot cd to an artifact_dir but I'm not sure that's the issue. By default we get RX read and execute access which almost always seems to work fine. If I remove RX by resetting the ACLs manually we get exactly the issue seen here.
using Pkg
# Using isoband since it has only one tiny DLL
Pkg.add(name="isoband_jll", version="0.2.3")
using isoband_jll: libisoband_path # loads the library
stat(libisoband_path) # mode: 0o100444 (-r--r--r--)
Sys.isexecutable(libisoband_path) # true
# RX - Read and execute access
run(`icacls $libisoband_path`) # Everyone:(RX,WA)
# Reset the ACL to lose the RX
run(`icacls $libisoband_path /reset`)
run(`icacls $libisoband_path`) # Everyone:(I)(R,W,D,DC)
stat(libisoband_path) # mode: 0o100444 (-r--r--r--)
Sys.isexecutable(libisoband_path) # false
# <restart julia>
using isoband_jll # Access is denied.
@giordano already linked to https://github.com/JuliaLang/Pkg.jl/pull/3349 above. If I call Pkg.set_readonly(libisoband_path) this retains the executable bits as desired, but perhaps it sometimes doesn't?
I copied an artifact_dir that wasn't working for experimentation, not realizing that this by default does not copy NTFS permissions, so I destroyed the evidence. Will be on the lookout.
Maybe it isn't executable? If that's the case and also the library isn't executable, that'd explain the issue
I have to say that in the case of Kaleido, running directly the executable using its full path works without access denied.
The problem there is that the kaleido.exe binary calls some dependencies with relative paths assuming that the program was called from the artifacts directory. That is why the included script (kaleido.cmd) to be called tries to set the working directory to the artifact dir (and fails because of access denied).
Kaleido seems to be a special case with a kaleido.cmd in the artifact dir root, that is why I think it is unrelated to this issue.
This is how you typically run a JLL ExecutableProduct:
using Kaleido_jll
run(`$(kaleido()) $arguments`)
This uses bin/kaleido.exe rather than ./kaleido.cmd.
Yeah I agree with yout @visr that Kaleido is weird and I also wish the library itself was built in a more standard way without requiring the current directory to be the artifact dir to work :(.
Trying to call kaleido the standard way you suggested throws an error due to this relative path issue:
julia> using Kaleido_jll
julia> run(`$(kaleido()) plotly`)
[0229/082015.195:ERROR:registration_protocol_win.cc(131)] TransactNamedPipe: The pipe has been ended. (0x6D)
[0229/082015.217:WARNING:resource_bundle.cc(435)] locale_file_path.empty() for locale
{"code": 0, "message": "Success", "result": null, "version": ""}
[0229/082015.642:ERROR:registration_protocol_win.cc(103)] CreateFile: The system cannot find the file specified. (0x2)
[0229/082016.956:ERROR:kaleido.cc(155)] Failed to find, or open, local file at ./js/kaleido_scopes.js with working directory C:\Users\Alberto.Mengali\Repos\github\others\PlotlyKaleido
I agree that kaleido specifically might be unrelated to the issue at hand, I found it weird that I can not cd in a folder anymore and the kaleido library not working anymore seems to be caused by a change in artifacts permissions defaults in Julia 1.10. It is a pity for kaleido specifically because that is the only way to export plots for any of the plotly based libraries :(.
It would seem that the issue with Kaleido specifically can be solved simply by explicitly setting the directory to call the program for using the related field of the Cmd command:
cmd = Cmd(`$(kaleido()) plotly`; dir = Kaleido_jll.artifact_dir)
This fixed the tests failing on https://github.com/JuliaPlots/PlotlyKaleido.jl/pull/17 even with the read-only permissions of 1.10 on windows (Interesting though that explicitly doing cd throws an error while passing the directory to Cmd does not)
Can someone icacls a directory that is not working?
Caught a few today on our CI servers. Here is one for libxml2 just like in the top post:
.\bin\libxml2-2.dll BUILTIN\Administrators:(R,WA)
DIRECTORY\Domain Users:(R,WA)
Everyone:(R,WA)
DIRECTORY\svc-teamcity-ansible:(R,WA)
NT AUTHORITY\SYSTEM:(F)
So indeed R read-only access, not RX read and execute access. I ran icacls on the whole directory with:
icacls aff35ec37f0361b8cfd28284a677e135a00d5f81 /t > icacls.txt
Results here: icacls.txt. So it looks like all files get (R,WA) and all directories (R,W,D,DC).
We worked around the issue with a manual icacls .julia\artifacts /reset /t.
read only access for QT5 DLL files on Windows, not read-and-execute. So those libraries cannot be opened (by QWT/Marble/MathGL/qwtw_jll/QWTWPlot in my case)
I am seeing this crop up in a package I maintain where we actually do want write access to our artifact (as an aside, is this very bad practice on our part? I'm not that familiar with Julia packaging practices, the feature was contributed by someone else originally)
as an aside, is this very bad practice on our part?
Yes, these artifacts are content-addressable storage, if you want to edit them the content changes and they lose their purpose. Consider using https://github.com/JuliaPackaging/Scratch.jl instead (depending on exactly what you want to do, but this is going off topic, consider asking for further help in https://discourse.julialang.org/)
I think we can ignore the bit about wanting to modify files in artifacts—we don't support that and don't want to. But what about the problems with loading libraries? Is the issue that the dll files need to have executable permissions in order to be loaded and lack them or that the directories need to have executable permissions and lack them? Is it clear to anyone what the actual problem here is?
I think the example in https://github.com/JuliaLang/julia/issues/52272#issuecomment-1969910902 shows that the issue is that the DLL files need to have executable permissions in order to be loaded, and for unknown reasons sporadically end up lacking them. https://github.com/JuliaLang/julia/issues/52272#issuecomment-1976855327 shows that when this permission change happens, the permissions for the entire artifact are modified, not just the DLL.
I'm not sure, but I suspect the permission change only ever happens when installing an artifact, not when loading an installed artifact. I tried to reproduce with removing and installing small artifacts in a loop, but couldn't trigger it.
I still maintain that Pkg.jl should never modify permissions on any files that it puts into a julia depot, permissions should always just be inherited. Instead it should modify the read-only attribute and leave the permissions entirely alone.
The thing is that once you modify any permission on a file, it no longer inherits any permissions from parent folders, and I'm sure that screws things up here. Just generally not a good idea to mess with permissions inside a user profile on Windows...
The set_readonly function here is just not what should be done on Windows by the package manager... On Windows that should just modify the file attribute and do nothing else.
But we have to mess with permissions to some extent because some files need to be executable while some files need to be not executable. We don't have to do set_readonly; it could just not be done on Windows, but it's good to ensure that packages and artifacts don't get accidentally modified and error instead if someone tries that. So what is the correct way to create files with various execute/read/write permissions on Windows?
But we have to mess with permissions to some extent because some files need to be executable while some files need to be not executable.
For what is worth, we already automatically set the executable bit in BinaryBuilder for shared libraries, so Pkg doesn't have to change anything. And this worked until Julia v1.9
However BinaryBuilder may set the permissions when constructing an artifact tarball, it's Tar that has to extract them on the client system. When installing a tarball, Tar has to set the executable bit. What I'm point out is that "never modify permissions on any files that it puts into a julia depot, permissions should always just be inherited" is not helpful guidance. If the more specific guidance is "don't try to make things read only on Windows", fine, we can change that, but that's a different thing. The whole Windows permissions thing is so nuts. I just want someone who actually understands Windows to tell us what the right way to do this is and then we can do it. But "don't ever change permissions" is not a workable answer.