WIP/RFC: Use CompilePreferences.jl to configure Python per project

[tkf]

This PR uses forthcoming CompilePreferences.jl stdlib https://github.com/JuliaLang/julia/pull/37595 to manage libpython setting for each project. The current design is to move deps/build.jl to a separate package PyPreferences.jl that gets a precompile cache for each configuration of libpython (python executable, using conda or not, PyJulia-mode or not, etc.). It is a separate package because we need to have a UI that can be imported even in a broken status (e.g., invalid python path).

Most of the logic lives in https://github.com/tkf/PyPreferences.jl but I thought it might be a good idea to open a PR to have a place to discuss the direction. At this point, this PR is still more or less for me to play with https://github.com/JuliaLang/julia/pull/37595 API to see if it is compatible with the usecase I have in mind. But feedback to the design/implementation is always welcome!

[tkf]

Here is an illustrative session (with a95568187a8b0d71b670b472eeeb3847e4f612c8 and https://github.com/tkf/PyPreferences.jl/commit/7006f6c18d7482eda9d1b093c531f1340423d08d). Suppose I have two projects:

shell> cat py37/Project.toml
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

shell> cat py38/Project.toml
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

Using PyPreferences to configure Python versions for each projects:

julia> using PyPreferences

(@v1.6) pkg> activate py37
 Activating environment at `~/.julia/dev/_wt/PyCall/preferences/tmp/py37/Project.toml`

julia> PyPreferences.use_system("python3.7")
PyPreferences.Implementations.PythonPreferences("python3.7", false, false)

(py37) pkg> activate py38
 Activating environment at `~/.julia/dev/_wt/PyCall/preferences/tmp/py38/Project.toml`

julia> PyPreferences.use_system("python3.8")
PyPreferences.Implementations.PythonPreferences("python3.8", false, false)

Now these projects contain compile-preferences table:

shell> cat py37/Project.toml

[compile-preferences.cc9521c6-0242-4dda-8d66-c47a9d9eec02]
python = "python3.7"
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

shell> cat py38/Project.toml

[compile-preferences.cc9521c6-0242-4dda-8d66-c47a9d9eec02]
python = "python3.8"
[deps]
PyCall = "438e738f-606a-5dbb-bf0a-cddfbfd45ab0"
PyPreferences = "cc9521c6-0242-4dda-8d66-c47a9d9eec02"

julia> exit()

These two projects use two different Python versions:

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py37
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.957 (2020-09-17)
 _/ |\__'_|_|_|\__'_|  |  preferences/e9e4d60854 (fork: 1 commits, 7 days)
|__/                   |

julia> using PyCall; PyCall.pyversion_build
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
v"3.7.8"

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py38
               _
   _       _ _(_)_     |  Documentation: https://docs.julialang.org
  (_)     | (_) (_)    |
   _ _   _| |_  __ _   |  Type "?" for help, "]?" for Pkg help.
  | | | | | | |/ _` |  |
  | | |_| | | | (_| |  |  Version 1.6.0-DEV.957 (2020-09-17)
 _/ |\__'_|_|_|\__'_|  |  preferences/e9e4d60854 (fork: 1 commits, 7 days)
|__/                   |

julia> using PyCall; PyCall.pyversion_build
[ Info: Precompiling PyCall [438e738f-606a-5dbb-bf0a-cddfbfd45ab0]
v"3.8.5"

I can use these two projects without re-compiling PyCall:

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py37
...

julia> using PyCall; println(pyimport("sys").version)  # no precompilation
3.7.8 (default, Aug  3 2020, 14:06:56)
[GCC 10.1.0]

$ ~/repos/watch/_wt/julia/preferences/usr/bin/julia --project=py38
...

julia> using PyCall; println(pyimport("sys").version)  # no precompilation
3.8.5 (default, Jul 27 2020, 08:42:51)
[GCC 10.1.0]

[stevengj]

Sounds good; I've been hoping for Pkg support for build preferences for a long time.

[tkf]

I suppose we'd want to have a shared/global preferences persists across Julia and PyCall versions (exactly like ~/.julia/prefs/PyCall) as a fallback of per-project preference, right? The current design of CompilePreferences.jl https://github.com/JuliaLang/julia/pull/37595 does not let us have such kind of preferences storage and I think it means that we need to implement something like ~/.julia/prefs/PyCall as a different mechanism (https://github.com/JuliaLang/julia/pull/37595#discussion_r495357958).

I think there is a way out for either directions in https://github.com/JuliaLang/julia/pull/37595 but I thought you might want to know this.

[PhilipVinc]

Hi @tkf, what's the status of this? Is there any agreement on how this should be implemented? I'd like to revive this and contribute to it because I would find it extremely useful.

[tkf]

I don't know if @stevengj looked into the implementation (as I posted it as a WIP), but I think we agree with the idea around this. The last time I tried, I didn't have time to integrate this into PyJulia.

If you can revive this again, I think it'd be great. It'd be extra awesome if you can tweak PyJulia simultaneously (or make it non-breaking for PyJulia). But otherwise, we probably need to temporarily break PyJulia after PyCall with the preference support is released.

Also, now that Pkg supports multiple packages in a single repository, I think it'd be easier to add lib/PyPreferences package in this repository.

[stevengj]

I would definitely like to switch PyCall to use the new Preferences architecture via https://github.com/JuliaPackaging/Preferences.jl … it doesn't seem like it needs a separate package anymore, so this PR should be reworked; might be easier to start a brand-new PR?

[tkf]

I don't mind if this PR is discarded, but I think it'd be better to separate the preference package. This is because it'd be much easier if the user can set the preference without loading PyCall itself. Without a separate package, we'd need to be able to make PyCall loadable in a "broken" state (i.e., cannot find an appropriate libpython). It's technically possible but I'm strongly against this approach since it'd require us to test PyCall twice if we want to be really sure that it works in both states. Furthermore, PyPreferences approach makes libpython discovery code (the scripts in deps) much easier to test, because we can run them in isolation without re-compiling PyCall.