JuliaLang
Downloads EASY_HOOK regression in v1.11.0?
I have this as my startup.jl file:
# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.
using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy.com:8080")
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
end
In 1.10.5 everything's fine:
julia> versioninfo()
Julia Version 1.10.5
Commit 6f3fdf7b36 (2024-08-27 14:19 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-15.0.7 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
JULIA_HISTORY = U:\julialog\julia_repl.jl
JULIA_NUM_THREADS = 4
(@v1.10) pkg> up
Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Project.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.10\Manifest.toml`
In 1.11.0, it's not, trying to add anything that has an artifact will fail.
julia> versioninfo()
Julia Version 1.11.0
Commit 501a4f25c2 (2024-10-07 11:40 UTC)
Build Info:
Official https://julialang.org/ release
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: 4 × Intel(R) Xeon(R) Gold 6326 CPU @ 2.90GHz
WORD_SIZE: 64
LLVM: libLLVM-16.0.6 (ORCJIT, cascadelake)
Threads: 4 default, 0 interactive, 2 GC (on 4 virtual cores)
Environment:
JULIA_DEPOT_PATH = E:\Program Files\JuliaDepot
JULIA_HISTORY = U:\julialog\julia_repl.jl
JULIA_NUM_THREADS = 4
(@v1.11) pkg> up
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`
(@v1.11) pkg> add DuckDB
Resolving package versions...
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
Installed DuckDB_jll ─ v1.1.0+0
Installed DuckDB ───── v1.1.0
Failure artifact: DuckDB
Failure artifact: DuckDB
ERROR: Unable to automatically download/install artifact 'DuckDB' from sources listed in 'E:\Program Files\JuliaDepot\packages\DuckDB_jll\IFrj8\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/96856948f83b372315f21905a5e7fec7c380b2ad
- https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz
Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/DuckDB_jll.jl/releases/download/DuckDB-v1.1.0+0/DuckDB.v1.1.0.x86_64-w64-mingw32-cxx11.tar.gz
Doing something like
Downloads.EASY_HOOK[] = (easy, info) -> begin
println("RUNNING EASY HOOK!!!")
end
I get
(@v1.11) pkg> add Plots
RUNNING EASY HOOK!!!
Updating registry at `~/.julia/registries/General.toml`
RUNNING EASY HOOK!!!
Resolving package versions...
RUNNING EASY HOOK!!!
RUNNING EASY HOOK!!!
so the hook is at least running...
Changed it to:
# This file should contain site-specific commands to be executed on Julia startup;
# Users may store their own personal commands in `~/.julia/config/startup.jl`.
println("Defo read startup file!")
using Downloads
Downloads.DOWNLOADER[] = nothing
Downloads.EASY_HOOK[] = (easy, info) -> begin
println("RUNNING EASY HOOK!!!")
end
PS E:\Program Files\julia-1.11.0\bin> ./julia
Defo read startup file!
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.0 (2024-10-07)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.11) pkg> add Plots
┌ Warning: could not download https://pkg.julialang.org/registries
│ exception = RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/registries
└ @ Pkg.Registry E:\Program Files\julia-1.11.0\share\julia\stdlib\v1.11\Pkg\src\Registry\Registry.jl:77
Resolving package versions...
Failure artifact: x265
Failure artifact: x265
ERROR: Unable to automatically download/install artifact 'x265' from sources listed in 'E:\Program Files\JuliaDepot\packages\x265_jll\9auB3\Artifacts.toml'.
Sources attempted:
- https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://pkg.julialang.org/artifact/da2e0ac84abbe712490f8ee73aca3d512f1eaaab
- https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz
Error: RequestError: HTTP/1.1 407 Proxy Authentication Required (CONNECT tunnel failed, response 407) while requesting https://github.com/JuliaBinaryWrappers/x265_jll.jl/releases/download/x265-v3.5.0+0/x265.v3.5.0.x86_64-w64-mingw32.tar.gz
Diff between Downloads.jl in v1.10.5 vs v1.11.0 is https://github.com/JuliaLang/Downloads.jl/compare/ead289a7f1be7689738aaea9b12d919424a106ef...89d3c7dded535a77551e763a437a6d31e4d9bf84, it shouldn't be difficult to bisect (assuming the issue is there and not somewhere else).
Can you try load Pkg, set the hook through Pkg.Downloads and use the Pkg commands (not the REPL mode) to add the package?
@KristofferC It works fine if done like that:
julia> using Pkg
julia> Downloads.DOWNLOADER[] = nothing
julia> Downloads.EASY_HOOK[] = (easy, info) -> begin
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXY, "http://madeupproxy:8080")
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYUSERPWD, ":")
Downloads.Curl.setopt(easy, Downloads.Curl.CURLOPT_PROXYAUTH, Downloads.Curl.CURLAUTH_NTLM)
end
#3 (generic function with 1 method)
julia> Pkg.update()
Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`
julia> Pkg.add("Plots")
Resolving package versions...
Downloaded artifact: x265
Downloaded artifact: GR
Downloaded artifact: LERC
Downloaded artifact: XZ
Downloaded artifact: fzf
In fact, it works fine, just going:
julia> using Pkg
julia> Pkg.update()
Updating registry at `E:\Program Files\JuliaDepot\registries\General.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Project.toml`
No Changes to `E:\Program Files\JuliaDepot\environments\v1.11\Manifest.toml`
It's just going via the REPL where it gets upset.
So this seems to be a require_stdlib issue, but I've not been able to replicate it
With ~/.julia/config/startup.jl or etc/julia/startup.jl containing the above test code, the easy hook is run properly after a pkg repl switch (loading Pkg via require_stdlib)
julia % julia +1.11
Defo read startup file!
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.11.1 (2024-10-16)
_/ |\__'_|_|_|\__'_| | Official https://julialang.org/ release
|__/ |
(@v1.11) pkg> up
RUNNING EASY HOOK!!!
Updating registry at `~/.julia/registries/General.toml`
...
Perhaps this is a windows-specific issue.
That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state
Often a stdlib ends up pre-compiled for an unrelated project/workflow, and then the pre-compiled stdlib starts to load as a duplicate version everywhere
You have to check via Base.loaded_modules_array() whether a duplicate copy was loaded, in which case I expect you'll hit this MWE even on Linux
The premise of require_stdlib is that normal code does not interact with any of the side-loaded stdlibs (they operate in isolation), which this kind of "public global" is incompatible with
require_stdlib makes it illegal for stdlibs to have "public globals" like DOWNLOADER
I very much appreciate you being enlightened about this. So removing use of "public globals" is the fix?
One fix would be to move the globals in question into Base, where uniqueness is still a thing
Yeah.. I'm not happy about the restrictions either TBH. I think they're a rough combo of unintuitive, hard to enforce, and lead to rare bugs (that they are rare is worse, not better IMO) but it's the situation we have right now
Moving the globals to Base should fix the issue
FWIW, I don't really know why we added this behavior to require_stdlib in the first place.
It might be worth revisiting?
IIUC at least one of the reasons was to ensure that if the user wants to use a newer stdlib version than was bundled, say Pkg, we can still load the version that the repl expects, to avoid breakage.
It has been a bumpy road for sure..
Oof. Moving these into Base is going to be non-trivial..
default_downloader! and DOWNLOADER[] both use types from Curl.jl
You can't update DOWNLOADER[] for two copies of Downloads.jl at once if they have different Downloader types...
For Pkg, if you update it, I would want the repl to be the updated one. We just have to fix the common load failures from out of date manifests.
We should probably fork this discussion to a dedicated issue, or a call, but for completeness.
require_stdlib was added in https://github.com/JuliaLang/julia/pull/53326 which focused on fixing https://github.com/JuliaLang/julia/issues/53365
That's because whether require_stdlib loads a duplicate copy of a package or not depends on your compile cache state
This is fixed by https://github.com/JuliaLang/julia/pull/55908, which was marked for backporting to avoid this issue
