Unable to specify compat for BuildDependency

[imciner2]

For normal Dependency() items, you can specify a compat entry to use when building and installing the package using the compat keyword argument, but there is no similar method to constrain a BuildDependency() to use only a specific version of a package when building.

My use case for this is that I want to keep the versions of the MKL and MKL headers packages in alignment with each other to ensure that I build with the correct headers for the MKL library. To do this, I tried adding

# Dependencies that must be installed before this package can be built
dependencies = Dependency[
    # There isn't really a need to have this exact MKL version,
    # we just specify it manually to ensure that we get the same
    # version for the headers and library.
    Dependency("MKL_jll"; compat="2022.0.0"),
    BuildDependency("MKL_Headers_jll"; compat="2022.0.0"),
]

but it throws this error:

ERROR: LoadError: MethodError: no method matching BuildDependency(::String; compat="2022.0.0")
Closest candidates are:
  BuildDependency(::AbstractString; platforms) at /home/b63738im/.julia/packages/BinaryBuilderBase/IKZGk/src/Dependencies.jl:128 got unsupported keyword argument "compat"
  BuildDependency(::Pkg.Types.PackageSpec; platforms) at /home/b63738im/.julia/packages/BinaryBuilderBase/IKZGk/src/Dependencies.jl:125 got unsupported keyword argument "compat"
Stacktrace:
 [1] kwerr(::NamedTuple{(:compat,), Tuple{String}}, ::Type, ::String)
   @ Base ./error.jl:157
 [2] top-level scope
   @ ~/dev/julia/Yggdrasil/O/OSQP/OSQP_MKL/build_tarballs.jl:16
in expression starting at /home/b63738im/dev/julia/Yggdrasil/O/OSQP/OSQP_MKL/build_tarballs.jl:16

[giordano]

Compat bounds don't make much sense for build dependencies as that's specific to the generated JLL package. You can fix a specific version number by using a PackageSpec for the dependency

[imciner2]

The problem I am talking about is how do you specify the version of the BuildDependency that the JLL is built with. Right now, I don't see a way to do that. When you specify a compat bound on a normal Dependency, it will also respect that bound when choosing the JLL package to use at build time as well as at runtime for the JLL package. I think there should be an equivalent way of specifying valid pinned versions for a buildtime-only dependency.

[giordano]

As I said, by using a PackageSpec:

using BinaryBuilder, Pkg
mkl_version = v"2022.0.0"
dependencies = [
    Dependency("MKL_jll"; compat="$(mkl_version)"),
    BuildDependency(PackageSpec(; name="MKL_Headers_jll", version=mkl_version)),
]

Perhaps we could have a second optional argument for the build version like for Dependency, but I'm strongly against adding a compat keyword argument: that's not useful for non-runtime dependencies.

[maleadt]

I'm strongly against adding a compat keyword argument: that's not useful for non-runtime dependencies.

A case where this would be useful: CUDA-dependent packages generally only build against a major/minor version combination (e.g. "this library should be build against CUDA 11.8"). It's currently required to specify the full version in the Yggdrasil build recipes, i.e., BuildDependency(PackageSpec(; version=v"11.8.89")), which is not useful: The APIs are guaranteed to be identical, and the bug fixes that are put in patch versions are only in the libraries that will be provided by a runtime dependency (CUDA_Runtime_jll). Instead, IIUC, every CUDA recipe now has to embed full versions.

[kalmarek]

@giordano hmm, when specifying

dependencies = [
    BuildDependency(PackageSpec(;name="MKL_jll", version="2022.2")),
]

(same happens for version=v"2022.2.0")

BB/Pkg throws an error at me:

   Resolving package versions...
ERROR: LoadError: MethodError: no method matching length(::Pkg.Versions.VersionSpec)

Closest candidates are:
  length(::Union{Base.KeySet, Base.ValueIterator})
   @ Base abstractdict.jl:58
  length(::Union{LinearAlgebra.Adjoint{T, S}, LinearAlgebra.Transpose{T, S}} where {T, S})
   @ LinearAlgebra ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/LinearAlgebra/src/adjtrans.jl:295
  length(::Union{SparseArrays.FixedSparseVector{Tv, Ti}, SparseArrays.SparseVector{Tv, Ti}} where {Tv, Ti})
   @ SparseArrays ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/SparseArrays/src/sparsevector.jl:95
  ...

Stacktrace:
  [1] union!(s::Set{Any}, itr::Pkg.Versions.VersionSpec)
    @ Base ./abstractset.jl:102
  [2] intersect!(s::Set{Any}, itr::Pkg.Versions.VersionSpec)
    @ Base ./abstractset.jl:193
  [3] _shrink(shrinker!::typeof(intersect!), itr::String, itrs::Tuple{Pkg.Versions.VersionSpec})
    @ Base ./array.jl:2788
  [4] intersect(itr::String, itrs::Pkg.Versions.VersionSpec)
    @ Base ./array.jl:2792
  [5] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::Nothing, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:388
  [6] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1356
  [7] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::Nothing, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1328
  [8] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1366
  [9] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1383
 [10] add
    @ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372 [inlined]
 [11] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
 [12] Pkg_add(::Pkg.Types.Context, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:platform, :io), Tuple{Platform, Base.TTY}}})
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/ilpz3/src/Prefix.jl:610
 [13] (::BinaryBuilderBase.var"#95#101"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}})()
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/ilpz3/src/Prefix.jl:678
 [14] activate(f::BinaryBuilderBase.var"#95#101"{Bool, Prefix, Vector{Pkg.Types.PackageSpec}, Platform, Vector{String}, Vector{String}}, new_project::String)
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:1847
 [15] setup_dependencies(prefix::Prefix, dependencies::Vector{Pkg.Types.PackageSpec}, platform::Platform; verbose::Bool)
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/ilpz3/src/Prefix.jl:669
 [16] setup_dependencies
    @ ~/.julia/packages/BinaryBuilderBase/ilpz3/src/Prefix.jl:639 [inlined]
 [17] (::BinaryBuilder.var"#setup_deps#27")(f::typeof(BinaryBuilderBase.is_target_dependency), prefix::Prefix, dependencies::Vector{BuildDependency}, platform::Platform, verbose::Bool)
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/0CUml/src/AutoBuild.jl:790
 [18] autobuild(dir::AbstractString, src_name::AbstractString, src_version::VersionNumber, sources::Vector{<:BinaryBuilderBase.AbstractSource}, script::AbstractString, platforms::Vector, products::Vector{<:Product}, dependencies::Vector{<:BinaryBuilderBase.AbstractDependency}; verbose::Bool, debug::Bool, skip_audit::Bool, ignore_audit_errors::Bool, autofix::Bool, code_dir::Union{Nothing, String}, require_license::Bool, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/0CUml/src/AutoBuild.jl:793
 [19] build_tarballs(ARGS::Any, src_name::Any, src_version::Any, sources::Any, script::Any, platforms::Any, products::Any, dependencies::Any; julia_compat::String, kwargs::Base.Pairs{Symbol, V, Tuple{Vararg{Symbol, N}}, NamedTuple{names, T}} where {V, N, names, T<:Tuple{Vararg{Any, N}}})
    @ BinaryBuilder ~/.julia/packages/BinaryBuilder/0CUml/src/AutoBuild.jl:347

Is this a bug (here or in Pkg?), or am I using it wrong?

[giordano]

It used to work for example in https://github.com/JuliaPackaging/Yggdrasil/blob/3e16be470aca56d0831fb246e9f6fa853e90a0c2/L/LLVMExtra/build_tarballs.jl#L70 Is that Julia v1.9? Can you try with v1.7 (what is used here on CI)? If that works and v1.9 doesn't, it might be a regression in Pkg.

[maleadt]

https://github.com/JuliaPackaging/Yggdrasil/pull/6709#issuecomment-1549484146 is probably related.

[kalmarek]

@giordano it still happens on julia-1.7;

ERROR: LoadError: MethodError: no method matching length(::Pkg.Versions.VersionSpec) happens when I pass string to version. It doesn't there if I pass v"2022.2".

[giordano]

Yeah, a VersionNumber is supposed to work

[kalmarek]

so is it a bug in Pkg that cannot handle resolution with all possible PackageSpec?

[giordano]

I don't know, I don't understand anything in Pkg

[kalmarek]

@giordano here's a mnwe:

using Pkg
using BinaryBuilder

name = "SomeName"
version = v"1.0.0"

# Collection of sources required to build
sources = GitSource[
]

# Bash recipe for building across all platforms
script = raw"""

"""

# These are the platforms we will build for by default, unless further
# platforms are passed in on the command line
platforms = [
    Platform("x86_64", "linux"; libc="glibc"),
]

# The products that we will ensure are always built
products = Product[
]

# Dependencies that must be installed before this package can be built
dependencies = [
    BuildDependency(PackageSpec(;name="MKL_jll", version="2022.2")),
]

# Build the tarballs, and possibly a `build.jl` as well
build_tarballs(ARGS, name, version, sources, script, platforms, products, dependencies, julia_compat="1.6")

• It fails on julia-1.7 with the above error
• It works on julia-1.7 with version=v"2022.2"
• It fails on julia-1.9 with similar error related to length
• It fails on julia-1.9 with version=v"2022.2" with KeyError:

ERROR: LoadError: KeyError: key v"2022.2.0" not found
Stacktrace:
  [1] getindex
    @ ./dict.jl:484 [inlined]
  [2] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::Nothing, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:446
  [3] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1356
  [4] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::Nothing, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1328
  [5] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::Nothing)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1366
  [6] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1383
  [7] add
    @ ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372 [inlined]
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.0+0.x64.linux.gnu/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
  [9] Pkg_add(::Pkg.Types.Context, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:platform, :io), Tuple{Platform, Base.TTY}}})
    @ BinaryBuilderBase ~/.julia/packages/BinaryBuilderBase/ilpz3/src/Prefix.jl:610

[marcom]

Just ran into this problem, here are the failing and working combinations:

On Julia 1.9

    BuildDependency(PackageSpec(name="Readline_jll", version="8.1.1")),      # fail
    BuildDependency(PackageSpec(name="Readline_jll", version="8.1.1+0")),    # fail
    BuildDependency(PackageSpec(name="Readline_jll", version=v"8.1.1")),     # fail
    BuildDependency(PackageSpec(name="Readline_jll", version=v"8.1.1+0")),   # works

On Julia 1.7

    BuildDependency(PackageSpec(name="Readline_jll", version="8.1.1")),      # fail
    BuildDependency(PackageSpec(name="Readline_jll", version="8.1.1+0")),    # fail
    BuildDependency(PackageSpec(name="Readline_jll", version=v"8.1.1")),     # works
    BuildDependency(PackageSpec(name="Readline_jll", version=v"8.1.1+0")),   # works