`Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6")` doesn't install anything

[giordano]

Trying to make BinaryBuilderBase work with Julia nightly I found that Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6") does nothing:

julia> using Pkg, UUIDs

(@v1.8) pkg> activate --temp
  Activating new project at `/tmp/jl_KrcKAp`

julia> Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6")
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
    Updating `/tmp/jl_KrcKAp/Project.toml`
  [781609d7] + GMP_jll v6.2.0+5
    Updating `/tmp/jl_KrcKAp/Manifest.toml`
  [56f22d72] + Artifacts
  [8f399da3] + Libdl
  [781609d7] + GMP_jll v6.2.0+5

If I use Pkg.Operations.find_installed to get the path where v6.2.0+5 is installed I get

julia> Pkg.Operations.find_installed("GMP_jll", UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), Base.SHA1("40388878122d491a2e55b0e730196098595d8a90"))
"/home/mose/.julia/packages/GMP_jll/IA6Pi"

but this directory doesn't exist:

shell> ls $(Pkg.Operations.find_installed("GMP_jll", UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), Base.SHA1("40388878122d491a2e55b0e730196098595d8a90")))
ls: cannot access '/home/mose/.julia/packages/GMP_jll/IA6Pi': No such file or directory

shell> ls -l ~/.julia/packages/GMP_jll/
total 4
drwxr-xr-x 3 mose mose 4096 Jan  7 19:32 i3hkK

The only directory in packages/GMP_jll is 6.1.2+6:

julia> Pkg.Operations.find_installed("GMP_jll", UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), Base.SHA1("15abc5f976569a1c9d651aff02f7222ef305eb2a"))
"/home/mose/.julia/packages/GMP_jll/i3hkK"

shell> ls -l $(Pkg.Operations.find_installed("GMP_jll", UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), Base.SHA1("15abc5f976569a1c9d651aff02f7222ef305eb2a")))
total 28
-r--r--r-- 1 mose mose 8889 Jan  7 19:32 Artifacts.toml
-r--r--r-- 1 mose mose 1545 Jan  7 19:32 LICENSE
-r--r--r-- 1 mose mose  333 Jan  7 19:32 Project.toml
-r--r--r-- 1 mose mose 3222 Jan  7 19:32 README.md
drwxr-xr-x 3 mose mose 4096 Jan  7 19:32 src

[giordano]

For the record, it looks like in BinaryBuilder we're going to sidestep this issue by not using Pkg.add(; julia_version), but the fact that this doesn't work still stands.

[IanButterworth]

by not using Pkg.add(; julia_version)

Given that exists for BB, if it's not needed perhaps it could be torn out?

[giordano]

It turns out that fixing this bug would still be useful to workaround #2942

[fingolfin]

It sounds as to me as that it's not "needed" in BB because it's broken, and BB works around it; so perhaps if it was fixed, BB would again want to use it? @giordano ?

[giordano]

I don't think the current implementation of BinaryBuilderBase.setup_dependencies works around this bug, Elliot simply came up with a more straightforward way to install the packages, but, as shown by #2942, being able to install packages with Pkg.add([PackageSpec("GMP_jll")]; julia_version=v"1.6") would still be useful in its own right.

[giordano]

To add a data point, julia_version may not be relevant. Running this script in a clean depot:

using  Pkg, UUIDs

name = "GMP_jll"
uuid = UUID("781609d7-10c4-51f6-84f2-b8444358ff6d")
tree_hash = Base.SHA1("40388878122d491a2e55b0e730196098595d8a90")

Pkg.add([PackageSpec(; name, uuid, tree_hash)])
pkg_dir = Pkg.Operations.find_installed(name, uuid, tree_hash)

@show readdir(pkg_dir)

I get

  Installing known registries into `/tmp/tmp.kqDfoYmeJH`
    Updating registry at `/tmp/tmp.kqDfoYmeJH/registries/General.toml`
   Resolving package versions...
    Updating `/tmp/tmp.kqDfoYmeJH/environments/v1.7/Project.toml`
  [781609d7] + GMP_jll
    Updating `/tmp/tmp.kqDfoYmeJH/environments/v1.7/Manifest.toml`
  [56f22d72] + Artifacts
  [8f399da3] + Libdl
  [781609d7] + GMP_jll
Precompiling project...
  1 dependency successfully precompiled in 1 seconds
ERROR: LoadError: IOError: readdir("/tmp/tmp.kqDfoYmeJH/packages/GMP_jll/IA6Pi"): no such file or directory (ENOENT)
Stacktrace:
 [1] uv_error
   @ ./libuv.jl:97 [inlined]
 [2] readdir(dir::String; join::Bool, sort::Bool)
   @ Base.Filesystem ./file.jl:851
 [3] readdir(dir::String)
   @ Base.Filesystem ./file.jl:846
 [4] top-level scope
   @ show.jl:1047
in expression starting at /tmp/foo.jl:10

The same happens with

name = "LibOSXUnwind_jll"
uuid = UUID("a83860b7-747b-57cf-bf1f-3e79990d037f")
tree_hash = Base.SHA1("4ad86f1d058c211e8916ceddf2404ea75b002e47")

which is the problem reported in https://github.com/JuliaLang/Pkg.jl/issues/2942#issuecomment-1018606329

[staticfloat]

I'm still not 100% certain this is the wrong behavior. It depends on what Pkg.add() means. From a manifest standpoint, this is the correct behavior. If you were on Julia v1.6 and you did Pkg.add("GMP_jll"), the correct behavior would be to mark down that a GMP_jll stdlib has been added to the current project, and then do nothing afterwards. If you were on Julia v1.5, the correct behavior would be to actually install something, because GMP_jll isn't an stdlib on that version.

I think it's reasonable for BB to be required to jump through extra hoops in order to get stdlib JLLs. The purpose of Pkg.add(; julia_version) was to emulate what would happen if you did a Pkg.add() on a different Julia version. Since neither v1.6 nor v1.7 would install GMP_jll, that stays constant. The issue is that, for BB, we really actually want to get the bits that the old version has bundled within it. But I think that's a reasonable thing for BB to need to do.

The extent of Pkg's responsibility here, IMO, is to know which versions of which stdlibs belong to which versions of Julia, so that it can write out the manifest properly.

BB's responsibility is then to use that information to go and grab the necessary package versions and get those artifacts, using Pkg.add(; julia_version=nothing).

Pinging @KristofferC to see if he agrees with my delineation of where Pkg's responsibility starts and ends.

[staticfloat]

As an example of how this gets worked around in BinaryBuilder:

using Pkg, BinaryBuilderBase

verbose = true
dependencies = [PackageSpec(;name="GMP_jll")]
julia_version = v"1.6"
ctx = Pkg.Types.Context(;julia_version)
Pkg.API.handle_package_input!.(dependencies)
Pkg.add(ctx, dependencies)

for dep in dependencies
    dep.version = BinaryBuilderBase.stdlib_version(dep.uuid, julia_version)
    Pkg.Operations.load_tree_hash!(ctx.registries, dep, nothing)
    stdlib_pkgspec = BinaryBuilderBase.get_addable_spec(dep.name, dep.version; verbose)
    Pkg.add(ctx, [stdlib_pkgspec]; julia_version=nothing)

    run(`ls -la $(Pkg.Operations.find_installed(dep.name, dep.uuid, dep.tree_hash))`)
end

[KristofferC]

If I use Pkg.Operations.find_installed to get the path where v6.2.0+5 is installed I get

julia> Pkg.Operations.find_installed("GMP_jll", UUID("781609d7-10c4-51f6-84f2-b8444358ff6d"), Base.SHA1("40388878122d491a2e55b0e730196098595d8a90")) "/home/mose/.julia/packages/GMP_jll/IA6Pi"

but this directory doesn't exist:

This is not surprising since the find_installed call there is just a computation based on the inputs. You could run that on any machine, with whatever is installed and you would get the same answer.