PythonCall.jl icon indicating copy to clipboard operation
PythonCall.jl copied to clipboard
verified publisher icon JuliaPy

"Unsatisfiable Requirement" edge case error when `juliacall` is upgraded but Julia registries are not

Open brian-dellabetta opened this issue 2 years ago • 7 comments

Hello, I have a potential issue originally posted as a Discussion a few weeks back. I haven't gotten any responses, so I'm moving it to an issue in hopes of getting more traction. If my solution of adding another env var that, if set, calls Pkg.registry.update() inside the initialization script seems reasonable, I can try my hand at a PR.

Discussed in https://github.com/JuliaPy/PythonCall.jl/discussions/422

Originally posted by brian-dellabetta November 9, 2023 Hi there, we have come across an edge case when upgrading PythonCall/juliacall, and I am curious if anyone knows of a workaround or clean solution.

Initial Condition:

  • User has juliacall==0.9.14 installed in their Python env
  • User has [email protected] installed in the Julia project that juliacall links to.
  • User has not updated their julia registry to know that a new version of [email protected] available.

Steps to reproduce:

  • User does pip install juliacall==0.9.15
  • User runs from juliacall import Main as jl This will result in the following error:
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 ├─restricted to versions * by <redacted>, leaving only versions: 0.1.0-0.9.14
 │ └─<redacted> log:
 │   ├─possible versions are: 0.1.0 or uninstalled
 │   └─<redacted> is fixed to version 0.1.0
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1357
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1346
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1367
  [8] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1407
  [9] develop
    @ ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1399 [inlined]
 [10] develop(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; shared::Bool, preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:222
 [11] develop(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] develop(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/environments/pyjuliapkg/pyjuliapkg/install/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1

All I need to do is run jl.seval("import Pkg; Pkg.registry.update()") but I can't get access to jl, I hit the error above first. Any suggestions on how to programmatically resolve this, ideally in Python? Hoping there's a way without having to edit juliacall source code to allow for an init/startup script. If that is the case, I can create an issue and take a shot at a PR. It may be as simple as adding another env var option that injects Pkg.registry.update() in the init script somewhere around this line.

Thanks!

brian-dellabetta avatar Nov 27 '23 16:11 brian-dellabetta

I'd be OK with adding an unconditional Pkg.registry.update() in the right place. I thought Julia always updated the registries the first time you Pkg.add anything though??

cjdoris avatar Nov 29 '23 09:11 cjdoris

Thanks @cjdoris , I checked in julia and ] add PythonCall does in fact trigger a registry update even if PythonCall already exists in the project.

This may really be an issue for juliapkg, I noticed that PythonCall's version gets pinned in the juliapkg.json, but it is very strange that this is occurring:

[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.3 at /home/<redacted>/.julia/environments/pyjuliapkg/pyjuliapkg/install/bin/julia
[juliapkg] Using Julia project at /home/<redacted>/.julia/environments/pyjuliapkg
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.develop([<redacted>])
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
           julia> Pkg.precompile()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 <full error posted above>

I'm not entirely sure how juliapkg uses the .json files internally, maybe there is some subtlety. My concern with the explicit call to Pkg.registry.update() is that it should only occur in online mode, i.e. PYTHON_JULIAPKG_OFFLINE=no, but that would be using a juliapkg env var inside the juliacall source code.

Any thoughts? Next time a version of PythonCall/juliacall is released, I can check. Unfortunately this edge case is rather hard to reproduce.

brian-dellabetta avatar Nov 29 '23 20:11 brian-dellabetta

Actually this would be a change to juliapkg so no worries there.

It doesn't sound like that would fix things anyway, if the registry does update. You can roll back to an old version of the registry by cloning the registry repo and checking out an old commit. Can you do that and see if you can reproduce the issue?

cjdoris avatar Nov 30 '23 08:11 cjdoris

Sure, I will try this sometime next week

brian-dellabetta avatar Dec 01 '23 17:12 brian-dellabetta

Nm i had a chance to look just now. Steps to reproduce:

  1. Revert to old General registry commit
cd /tmp
git clone [email protected]:JuliaRegistries/General.git
cd General
git reset 46f90400dd 
git checkout -- . && git clean -fd
  1. Create conda environment, set up julia_env, and install PythonCall into julia_env
conda create -n pyjlcheck
conda activate pyjlcheck
pip install juliacall==0.9.14
python -c 'from juliacall import Main as jl'
  1. Set General registry to old cloned repo julia --project=~/anaconda3/envs/pyjlcheck/julia_env
] registry rm General
] registry add /tmp/General
  1. Pull latest changes from general registry and upgrade juliacall version
cd /tmp/General
git pull
pip install juliacall==0.9.15
  1. Run juliacall
from juliacall import Main as jl

errors out with full logs:

>>> from juliacall import Main as jl
[juliapkg] Locating Julia ^1.6.1
[juliapkg] Using Julia 1.9.4 at /Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia
[juliapkg] Using Julia project at /Users/brian/anaconda3/envs/pyjlcheck/julia_env
[juliapkg] Installing packages:
           julia> import Pkg
           julia> Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")])
           julia> Pkg.resolve()
   Resolving package versions...
ERROR: Unsatisfiable requirements detected for package PythonCall [6099a3de]:
 PythonCall [6099a3de] log:
 ├─possible versions are: 0.1.0-0.9.14 or uninstalled
 └─restricted to versions 0.9.15 by an explicit requirement — no versions left
Stacktrace:
  [1] check_constraints(graph::Pkg.Resolve.Graph)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:998
  [2] Pkg.Resolve.Graph(compat::Dict{Base.UUID, Dict{VersionNumber, Dict{Base.UUID, Pkg.Versions.VersionSpec}}}, compat_weak::Dict{Base.UUID, Dict{VersionNumber, Set{Base.UUID}}}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, verbose::Bool, julia_version::VersionNumber)
    @ Pkg.Resolve ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Resolve/graphtype.jl:345
  [3] deps_graph(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, uuid_to_name::Dict{Base.UUID, String}, reqs::Dict{Base.UUID, Pkg.Versions.VersionSpec}, fixed::Dict{Base.UUID, Pkg.Resolve.Fixed}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:587
  [4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, installed_only::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:407
  [5] targeted_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1362
  [6] tiered_resolve(env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, julia_version::VersionNumber, try_all_installed::Bool)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1351
  [7] _resolve(io::Base.TTY, env::Pkg.Types.EnvCache, registries::Vector{Pkg.Registry.RegistryInstance}, pkgs::Vector{Pkg.Types.PackageSpec}, preserve::Pkg.Types.PreserveLevel, julia_version::VersionNumber)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1372
  [8] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}, new_git::Set{Base.UUID}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform)
    @ Pkg.Operations ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1389
  [9] add
    @ ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/Operations.jl:1378 [inlined]
 [10] add(ctx::Pkg.Types.Context, pkgs::Vector{Pkg.Types.PackageSpec}; preserve::Pkg.Types.PreserveLevel, platform::Base.BinaryPlatforms.Platform, kwargs::Base.Pairs{Symbol, Base.TTY, Tuple{Symbol}, NamedTuple{(:io,), Tuple{Base.TTY}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:275
 [11] add(pkgs::Vector{Pkg.Types.PackageSpec}; io::Base.TTY, kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:156
 [12] add(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/share/julia/stdlib/v1.9/Pkg/src/API.jl:145
 [13] top-level scope
    @ none:1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 228, in <module>
    init()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliacall/__init__.py", line 153, in init
    CONFIG['exepath'] = exepath = juliapkg.executable()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 296, in executable
    resolve()
  File "/Users/brian/anaconda3/lib/python3.8/site-packages/juliapkg/deps.py", line 275, in resolve
    run([exe, '--project='+project, '--startup-file=no', '-e', '; '.join(script)], check=True)
  File "/Users/brian/anaconda3/lib/python3.8/subprocess.py", line 516, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia', '--project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env', '--startup-file=no', '-e', 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()']' returned non-zero exit status 1.

I can confirm that taking that command

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

fails but adding Pkg.Registry.update() works (though we'd probably only want to do this in online mode)

/Users/brian/.julia/juliaup/julia-1.9.4+0.x64.apple.darwin14/bin/julia --project=/Users/brian/anaconda3/envs/pyjlcheck/julia_env --startup-file=no -e 'import Pkg; Pkg.Registry.update(); Pkg.add([Pkg.PackageSpec(name="PythonCall", uuid="6099a3de-0909-46bc-b1f4-468b9a2dfc0d")]); Pkg.resolve()'

brian-dellabetta avatar Dec 01 '23 18:12 brian-dellabetta

@cjdoris should we just change it to auto-update registry? I can create a PR, I think it just amounts to changing this line to

script = ['import Pkg', 'Pkg.Registry.update()']

It's already in the if not STATE['offline']: check so we don't need to worry about offline mode

brian-dellabetta avatar Dec 06 '23 22:12 brian-dellabetta

PR submitted to resolve this in pyjuliapkg -- https://github.com/JuliaPy/pyjuliapkg/pull/22

brian-dellabetta avatar Dec 21 '23 13:12 brian-dellabetta