Tree hash mismatch error when installing on WSL

[MaximeBouton]

I am getting a Tree Hash Mismatch! error when trying to install SpecialFunctions.

Setup:

• julia 1.3 : fresh install, I just have Revise installed
• windows subsystem for linux (WSL)

Full error message with step to reproduce:

(v1.3) pkg> add SpecialFunctions
 Resolving package versions...
┌ Error: Tree Hash Mismatch!
│   Expected git-tree-sha1:   d322d52dbe634d659d67312eea8a78c16112127e
│   Calculated git-tree-sha1: d9046630e42d7bde08766dc041254b730e0110af
└ @ Pkg.Artifacts /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:724
ERROR: Unable to automatically install 'OpenSpecFun' from '/mnt/c/Users/Maxime/wsl/.julia/packages/OpenSpecFun_jll/XrUb6/Artifacts.toml'
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] #ensure_artifact_installed#42(::Pkg.BinaryPlatforms.Linux, ::Bool, ::typeof(Pkg.Artifacts.ensure_artifact_installed), ::String, ::Dict{String,Any}, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:810
 [3] (::Pkg.Artifacts.var"#kw##ensure_artifact_installed")(::NamedTuple{(:platform, :verbose),Tuple{Pkg.BinaryPlatforms.Linux,Bool}}, ::typeof(Pkg.Artifacts.ensure_artifact_installed), ::String, ::Dict{String,Any}, ::String) at ./none:0
 [4] #ensure_all_artifacts_installed#43(::Pkg.BinaryPlatforms.Linux, ::Nothing, ::Bool, ::Bool, ::typeof(Pkg.Artifacts.ensure_all_artifacts_installed), ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Artifacts.jl:852
 [5] #ensure_all_artifacts_installed at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:0 [inlined]
 [6] #download_artifacts#78(::Pkg.BinaryPlatforms.Linux, ::Bool, ::typeof(Pkg.Operations.download_artifacts), ::Array{String,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:580
 [7] #download_artifacts at ./none:0 [inlined]
 [8] #download_artifacts#73(::Pkg.BinaryPlatforms.Linux, ::Bool, ::typeof(Pkg.Operations.download_artifacts), ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:570
 [9] (::Pkg.Operations.var"#kw##download_artifacts")(::NamedTuple{(:platform,),Tuple{Pkg.BinaryPlatforms.Linux}}, ::typeof(Pkg.Operations.download_artifacts), ::Array{Pkg.Types.PackageSpec,1}) at ./none:0
 [10] #add#112(::Bool, ::Pkg.BinaryPlatforms.Linux, ::typeof(Pkg.Operations.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}, ::Array{Base.UUID,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/Operations.jl:1017
 [11] #add at ./none:0 [inlined]
 [12] #add#25(::Bool, ::Pkg.BinaryPlatforms.Linux, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Pkg.API.add), ::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:102
 [13] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/API.jl:72
 [14] do_add!(::Dict{Symbol,Any}, ::Array{Pkg.Types.PackageSpec,1}, ::Dict{Symbol,Any}) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/REPLMode.jl:505
 [15] #invokelatest#1(::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::typeof(Base.invokelatest), ::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:709
 [16] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:708
 [17] do_cmd!(::Pkg.REPLMode.Command, ::REPL.LineEditREPL) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/REPLMode.jl:412
 [18] #do_cmd#23(::Bool, ::typeof(Pkg.REPLMode.do_cmd), ::REPL.LineEditREPL, ::String) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/REPLMode.jl:391
 [19] do_cmd at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/REPLMode.jl:387 [inlined]
 [20] (::Pkg.REPLMode.var"#28#31"{REPL.LineEditREPL,REPL.LineEdit.Prompt})(::REPL.LineEdit.MIState, ::Base.GenericIOBuffer{Array{UInt8,1}}, ::Bool) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/Pkg/src/REPLMode.jl:619
 [21] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [22] invokelatest at ./essentials.jl:708 [inlined]
 [23] run_interface(::REPL.Terminals.TextTerminal, ::REPL.LineEdit.ModalInterface, ::REPL.LineEdit.MIState) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/REPL/src/LineEdit.jl:2306
 [24] run_frontend(::REPL.LineEditREPL, ::REPL.REPLBackendRef) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:1045
 [25] run_repl(::REPL.AbstractREPL, ::Any) at /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.3/REPL/src/REPL.jl:201
 [26] (::Base.var"#770#772"{Bool,Bool,Bool,Bool})(::Module) at ./client.jl:382
 [27] #invokelatest#1 at ./essentials.jl:709 [inlined]
 [28] invokelatest at ./essentials.jl:708 [inlined]
 [29] run_main_repl(::Bool, ::Bool, ::Bool, ::Bool, ::Bool) at ./client.jl:366
 [30] exec_options(::Base.JLOptions) at ./client.jl:304
 [31] _start() at ./client.jl:460

[simonbyrne]

@staticfloat any ideas?

[giordano]

What's your umask? This looks like a duplicate of https://github.com/JuliaPackaging/BinaryBuilder.jl/issues/527

[MaximeBouton]

This is the output:

$ umask
0000

[StefanKarpinski]

Doesn't umask 0o0000 force all files you create to have no permissions?

[StefanKarpinski]

Sorry, I forgot that the on bits are the ones that get turned off in the result. Not sure what's going on.

[staticfloat]

Windows Subsystem for Linux? Does your Julia report itself as running on Windows or running on Linux? What is your versioninfo()?

[staticfloat]

If it thinks its running on Linux, I bet that the branches we have for Sys.iswindows() don't get called, and that results in our windows-specific workarounds not being triggered.

[MaximeBouton]

It seems like its running on Linux:

julia> versioninfo()
Julia Version 1.3.0
Commit 46ce4d7933 (2019-11-26 06:09 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Core(TM) i7-6500U CPU @ 2.50GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.1 (ORCJIT, skylake)
Environment:
  JULIA_DEPOT_PATH = /mnt/c/Users/Maxime/wsl/.julia/
  JULIA_PKGDIR = /mnt/c/Users/Maxime/wsl/.julia/

[StefanKarpinski]

What does Sys.iswindows() return?

[staticfloat]

Yep; this is a problem. It thinks its linux and thereby thinks that the filesystem will behave in a Linux-like fashion (e.g. having reasonable Sys.isexecutable() behavior) but the underlying filesystem may not.

Could I ask you to do the following test for me:

using Pkg.Artifacts

function create_file(dir::String, exe::Bool = false)
    foo_path = joinpath(dir, "foo")
    open(foo_path, "w") do io
        println(io, "hello")
    end

    if exe
        chmod(foo_path, 0o775)
    end

    mode = string(stat(foo_path).mode, base=8)
    is_exe = Sys.isexecutable(foo_path)
    @info("make exectuable: $(exe)", mode, is_exe)
end

first_hash = create_artifact(dir -> create_file(dir, false))
@show first_hash

second_hash = create_artifact(dir -> create_file(dir, true))
@show second_hash

Running that script should give you an output such as the following:

┌ Info: make exectuable: false
│   mode = "100664"
└   is_exe = false
first_hash = SHA1("206c918ca6bcf822f8f0b4d63226aad05bf56067")
┌ Info: make exectuable: true
│   mode = "100775"
└   is_exe = true
second_hash = SHA1("4a8aac5671804b3504145f323061e9417566ea31")

[MaximeBouton]

julia> Sys.iswindows()
false

Here is the output of the script:

┌ Info: make exectuable: false
│   mode = "100777"
└   is_exe = true
first_hash = SHA1("4a8aac5671804b3504145f323061e9417566ea31")
┌ Info: make exectuable: true
│   mode = "100777"
└   is_exe = true
second_hash = SHA1("4a8aac5671804b3504145f323061e9417566ea31")

[staticfloat]

Yep; that's the problem. The filesystem is built upon a foundation of lies.

This is actually a really pathological case for us since even our intended designs to fix Sys.isexecutable() on Windows wouldn't work here, due to the fact that Julia thinks its running on Linux, and so wouldn't use the windows codepaths. The true problem here is that whatever filesystem translation layer is happening in the WSL land is making the same kind of sacrifice that many other Linux-on-Windows systems make, which is to pretend that executable permissions more or less do not exist.

The immediate workaround that I can think of is to have Pkg test to see if the filesystem being used for create_artifact() is "permissions-insensitive", and if so, engage the "ignore tree hashes" workaround. This would have the side-benefit of dodging future problems when someone tries to use this code on a really non-default configuration like Linux on some exotic filesystem that doesn't obey permissions.

The downside to these workarounds is that they only hide the problem; the fact of the matter is that Windows machines are "blind" to permissions differences, and so the hashes they calculate are different from those that the rest of the world calculates. This means that when attempting to create and bind an artifact on Windows, everything works until we try to share the artifact elsewhere. We are relying on the fact that most people are authoring on Linux, and we're just overriding the hash check on Windows. Not an ideal situation at all.

[staticfloat]

@MaximeBouton after reading up on the WSL filesystem setup, what happens if you set your JULIA_DEPOT_PATH to point to something that is mounted in one of the linux-native filesystems? I'm not sure why your julia package directory is pointing to /mnt/c/Users/Maxime/wsl/.julia, but can you have it point to something like /home/Maxime/.julia? Or maybe even just /tmp/.julia for testing, and see if you have the same issues?

[MaximeBouton]

It worked! Thanks @staticfloat ! I had changed the JULIA_DEPOT_PATH variable to point to a folder that I could access through the windows file explorer but I don't really need that actually.

edit: I changed it to /home/maxime/.julia, I also tried to not set anything for both JULIA_DEPOT_PATH and JULIA_PKG_DIR and it worked both time

[staticfloat]

@MaximeBouton is there any documentation you read to get Julia working on WSL? If so, we should add this little bit of wisdom into the instructions so that others know not to do what you did. :)

[MaximeBouton]

I have naively followed the linux installation instruction and it just worked before. I am not sure where the best place would be for a WSL specific documentation.

[marcopigg]

I am getting the same issue, trying to install Plots.jl on an external ssd drive in WSL. I understand the use case is limited, but having a working julia environment on an external drive allows me to avoid having to reinstall (and keep up to date) julia on multiple systems, such as:

• ubuntu workstation at home
• windows laptop
• windows PCs at my institution (where installing Linux is not an option).

I tried both versions 1.5.3 and the latest nightly. The issue appears on both.

I am having the issue on WSL 1 (to which I am tied because of WSL issue #5911). @MaximeBouton was the original issue on WSL 1 or WSL 2? If it was WSL 1, does anyone know if the issue is still there on WSL 2?

[giordano]

and the latest nightly

Really it doesn't work on nightly? Your issue sounds like https://github.com/JuliaLang/Pkg.jl/issues/2317, which should have been fixed

[marcopigg]

Ok sorry it does work now. I missed the JULIA_PKG_IGNORE_HASHES option.

[MaximeBouton]

@marcopigg it was on WSL 1, I have not tried on WSL 2 yet

[sapiomr]

Can you clarify exactly where you put the JULIA_PKG_IGNORE_HASHES = 1 flag? Is it ok to type this in the session or does it have to be appended to a specific sytem file?