Non-registered dependency not working with Julia 1.7

[cirocavani]

I am building 2 separated JLL, one for the C++ library and other for the CxxWrap glue-code library. Both packages are not registered and I am trying to build both in my machine. The problem I noticed was that when I use julia 1.7, the non-registered JLL dependency fails.

I will push this code soon.

But the error is simple,

build_tarballs.jl for CxxWrap glue-code library:

# Dependencies that must be installed before this package can be built
dependencies = [
#    Dependency("Box2D_jll", v"2.4.1"),
    Dependency(PackageSpec(;
        name = "Box2D_jll",
        uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834",
        # url = "https://github.com/cirocavani/Box2D_jll.jl",
        path = joinpath(Pkg.devdir(), "Box2D_jll"),
    )),
    Dependency("libcxxwrap_julia_jll"),
    BuildDependency("libjulia_jll"),
]

1.7 error:

[ Info: Building and deploying version 2.4.1+0 to /Users/cirocavani/.julia/dev/Box2D_Julia_jll
[ Info: Building for x86_64-apple-darwin
[ Info: Directory "./src" found
[ Info: Copying content of src in srcdir...
    Updating registry at `~/.julia/registries/General.toml`
   Resolving package versions...
ERROR: LoadError: expected package `Box2D_jll [6756b63d]` to be registered
Stacktrace:
  [1] pkgerror(msg::String)
    @ Pkg.Types /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Types.jl:68
  [2] check_registered
    @ /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1090 [inlined]
  [3] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1152
  [4] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::Nothing)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1125
  [5] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1160
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Platform)
    @ Pkg.Operations /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/Operations.jl:1176
  [7] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:268
  [8] (::BinaryBuilderBase.var"#88#94"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}})()
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/UqKXf/src/Prefix.jl:568
  [9] activate(f::BinaryBuilderBase.var"#88#94"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}}, new_project::String)
    @ Pkg.API /Applications/Julia-1.7.app/Contents/Resources/julia/share/julia/stdlib/v1.7/Pkg/src/API.jl:1599
 [10] setup_dependencies(prefix::Prefix, dependencies::Vector{Pkg.Types.PackageSpec}, platform::Platform; verbose::Bool)
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/UqKXf/src/Prefix.jl:561
 [11] (::BinaryBuilder.var"#setup_deps#27")(f::typeof(BinaryBuilderBase.is_target_dependency), prefix::Prefix, dependencies::Vector{BinaryBuilderBase.AbstractDependency}, platform::Platform, verbose::Bool)
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/Ftk2f/src/AutoBuild.jl:754
 [12] autobuild(dir::AbstractString, src_name::AbstractString, src_version::VersionNumber, sources::Vector{<:BinaryBuilderBase.AbstractSource}, script::AbstractString, platforms::Vector, products::Vector{<:Product}, dependencies::Vector{<:BinaryBuilderBase.AbstractDependency}; verbose::Bool, debug::Bool, skip_audit::Bool, ignore_audit_errors::Bool, autofix::Bool, code_dir::Union{Nothing, String}, require_license::Bool, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/Ftk2f/src/AutoBuild.jl:757
 [13] build_tarballs(ARGS::Any, src_name::Any, src_version::Any, sources::Any, script::Any, platforms::Any, products::Any, dependencies::Any; julia_compat::String, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/Ftk2f/src/AutoBuild.jl:321
 [14] top-level scope
    @ ~/Garage/Box2D_Julia_jll/build_tarballs.jl:94
in expression starting at /Users/cirocavani/Garage/Box2D_Julia_jll/build_tarballs.jl:94

1.6

[ Info: Building and deploying version 2.4.1+0 to /Users/cirocavani/.julia/dev/Box2D_Julia_jll
[ Info: Building for x86_64-apple-darwin
[ Info: Directory "./src" found
[ Info: Copying content of src in srcdir...
    Updating git-repo `/Users/cirocavani/.julia/dev/Box2D_jll`
   Resolving package versions...
    Updating `~/Garage/Box2D_Julia_jll/build/x86_64-apple-darwin/H6JjTr7P/x86_64-apple-darwin14-libgfortran4-cxx11/.project/Project.toml`
  [6756b63d] + Box2D_jll v2.4.1+0 `/Users/cirocavani/.julia/dev/Box2D_jll#main`
→ [3eaa8342] + libcxxwrap_julia_jll v0.9.0+1
→ [5ad3ddd2] + libjulia_jll v1.8.0+0
        Info packages marked with → not downloaded, use `instantiate` to download
    Updating `~/Garage/Box2D_Julia_jll/build/x86_64-apple-darwin/H6JjTr7P/x86_64-apple-darwin14-libgfortran4-cxx11/.project/Manifest.toml`
  [34da2185] + Compat v2.2.1
  [692b3bcd] + JLLWrappers v1.4.1
  [21216c6a] + Preferences v1.2.4
  [6756b63d] + Box2D_jll v2.4.1+0 `/Users/cirocavani/.julia/dev/Box2D_jll#main`
→ [3eaa8342] + libcxxwrap_julia_jll v0.9.0+1
→ [5ad3ddd2] + libjulia_jll v1.8.0+0
  [0dad84c5] + ArgTools v1.1.1
  [56f22d72] + Artifacts v1.3.0
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [8bb1440f] + DelimitedFiles
  [8ba89e20] + Distributed
  [f43a241f] + Downloads v1.6.0
  [7b1f6079] + FileWatching
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL v0.6.3
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [37e2e46d] + LinearAlgebra
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [a63ad114] + Mmap
  [ca575930] + NetworkOptions v1.2.0
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA v0.5.7
  [9e88b42a] + Serialization
  [1a1011a3] + SharedArrays
  [6462fe0b] + Sockets
  [2f01184e] + SparseArrays
  [10745b16] + Statistics
  [fa267f1f] + TOML v1.0.3
  [a4e569a6] + Tar v1.9.2
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll v7.81.0+0
  [29816b5a] + LibSSH2_jll v1.10.2+0
  [183b4373] + LibUV_jll v2.0.1+5
  [745a5e78] + LibUnwind_jll v1.5.0+1
  [c8ffd9c3] + MbedTLS_jll v2.28.0+0
  [14a3606d] + MozillaCACerts_jll v2022.2.1+0
  [83775a58] + Zlib_jll v1.2.12+2
  [8e850ede] + nghttp2_jll v1.41.0+2
  [3f19e933] + p7zip_jll v16.2.1+2
        Info packages marked with → not downloaded, use `instantiate` to download
┌ Warning: Dependency LibUnwind_jll does not have a mapping for artifact LibUnwind for platform x86_64-apple-darwin14-libgfortran4-cxx11
└ @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/IKZGk/src/Prefix.jl:507
...

[fcdimitr]

Any updates regarding this issue? I encounter the same problem on Julia v1.8.

[giordano]

Looks to me like a breaking change in Pkg, reproducer is:

using Pkg
Pkg.add(Pkg.Types.Context(), [PackageSpec(; name = "Box2D_jll", uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834", url = "https://github.com/cirocavani/Box2D_jll.jl")])

Please open an issue there.

[fcdimitr]

The proposed solution at https://github.com/JuliaLang/Pkg.jl/issues/3251 is to remove the context argument.

Indeed, this works:

Pkg.add([PackageSpec(; name = "Box2D_jll", uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834", url = "https://github.com/cirocavani/Box2D_jll.jl")])

@giordano I am not familiar with the context argument. Is this something that can be removed from the internal calls of BinaryBuilder.jl or not?

Thank you!

[giordano]

Is this something that can be removed from the internal calls of BinaryBuilder.jl or not?

No

[sloede]

Based on the suggestion in https://github.com/JuliaLang/Pkg.jl/issues/3251#issuecomment-1476330679, I found a workaround. Put the following code in your build_tarballs.jl for your unregistered dependency, and then it works (for me):

ps = PackageSpec(; name = "Box2D_jll", uuid = "6756b63d-2e54-5dc6-816b-1fc83e1ae834",
                   url = "https://github.com/cirocavani/Box2D_jll.jl")
Pkg.API.handle_package_input!(ps)

# Dependencies that must be installed before this package can be built
dependencies = [
    Dependency(ps)
    # other dependencies...
]

[IanButterworth]

It seems BB should do that instead. Even if it doesn't fix every issue, it's what Pkg does in every case via the public API, so I don't see why BB shouldn't do that (and a deepcopy first)