poetry not compatible with precompiled python

[andersfylling]

Describe the bug When I tried setting up python and poetry for a project, the poetry install fails with:

/home/anders/.local/share/mise/installs/poetry/1.8.2/venv/bin/python3: error while loading shared libraries: /home/anders/.local/share/mise/installs/poetry/1.8.2/venv/bin/../lib/libpython3.10.so.1.0: cannot open shared object file: No such file or directory

Traceback:

  File "<stdin>", line 923, in main
  File "<stdin>", line 562, in run

To Reproduce

1. mkdir test && cd test
2. mise u [email protected]
3. mise u poetry

This led to the above error. I have never had poetry installed on this machine before, nor mise.

Expected behavior That poetry successfully installed.

mise doctor output

version: 2024.2.19 linux-x64 (eae477c 2024-03-03)
activated: yes
shims_on_path: no

build_info: 
  Target: x86_64-unknown-linux-gnu
  Features: DEFAULT, NATIVE_TLS
  Built: Sun, 3 Mar 2024 06:11:29 +0000
  Rust Version: rustc 1.76.0 (07dca489a 2024-02-04) (Arch Linux rust 1:1.76.0-1)
  Profile: release

shell: 
  /bin/zsh
  zsh 5.9 (x86_64-pc-linux-gnu)

dirs: 
  data: ~/.local/share/mise
  config: ~/.config/mise
  cache: ~/.cache/mise
  state: ~/.local/state/mise
  shims: ~/.local/share/mise/shims

config_files: 
  ~/dev/test/.rtx.toml
  ~/dev/test/.mise.toml

plugins: 
  bun     (core)
  deno    (core)
  go      (core)
  gradle  https://github.com/rfrancis/asdf-gradle.git#ffc271c
  java    (core)
  node    (core)
  poetry  https://github.com/mise-plugins/mise-poetry.git#82d11e3
  python  (core)
  ruby    (core)

toolset: 
  [email protected]  

env_vars: 
  MISE_SHELL=zsh

settings: 
  activate_aggressive = false
  all_compile = false
  always_keep_download = false
  always_keep_install = false
  asdf_compat = false
  cargo_binstall = true
  color = true
  disable_default_shorthands = false
  disable_tools = []
  experimental = false
  go_default_packages_file = "~/.default-go-packages"
  go_download_mirror = "https://dl.google.com/go"
  go_repo = "https://github.com/golang/go"
  go_set_gopath = false
  go_set_goroot = true
  go_skip_checksum = false
  jobs = 4
  legacy_version_file = true
  legacy_version_file_disable_tools = []
  node_compile = false
  not_found_auto_install = true
  paranoid = false
  plugin_autoupdate_last_check_duration = "7d"
  python_compile = false
  python_default_packages_file = "/home/anders/.default-python-packages"
  python_pyenv_repo = "https://github.com/pyenv/pyenv.git"
  raw = false
  trusted_config_paths = []
  quiet = false
  verbose = false
  yes = false
  ci = false
  debug = false
  trace = false
  log_level = "info"
  python_venv_auto_create = false

  [status]
  missing_tools = "if_other_versions_installed"
  show_env = false
  show_tools = false

No problems found

Additional context OS: arch linux 6.7.7-arch1-1

edit by @jdx

precompiled pythons are not compatible with poetry. I don't know why but the issue needs to be fixed upstream since the issue is not with mise itself but the pythons mise uses. Here's how to disable precompiled pythons and reinstall:

mise settings set python_compile 1
mise install python -f

[wasdee]

same here, WSL2, windows freshly install

/home/xx/.local/share/mise/installs/poetry/1.8.2/venv/bin/python3: error while loading shared libraries: 

/home/xx/.local/share/mise/installs/poetry/1.8.2/venv/bin/../lib/libpython3.12.so.1.0: cannot open shared object file: No such file or directory

/home/din/.local/share/mise/installs/poetry/1.8.2/venv/bin/python3: error while loading shared libraries: /home/din/.local/share/mise/installs/poetry/1.8.2/venv/bin/../lib/libpython3.12.so.1.0: cannot open shared object file: No such file or directory

Traceback:

  File "<stdin>", line 923, in main
  File "<stdin>", line 562, in run

[wasdee]

workaround

• use python compile installation, set mise flag to compile python, reinstall with --force

[Nickersoft]

Can confirm am facing this same issue 😭 @wasdee What is the full command you're running as a workaround?

[wasdee]

Can confirm am facing this same issue 😭 @wasdee What is the full command you're running as a workaround?

suppose, I want to use 3.11.1. and i got a pre-compiled one installed. and already install pre-compile pkgs

$ mise settings set python_compile true

# maybe explicitly uninstall first
$ mise install [email protected] --force

$ mise use -g [email protected] 

$ mise  use -g poetry@latest

This is what i vaguely remember.

[mustafa0x]

same issue. i hope this can be fixed soon. as a workaround i installed pipx, then poetry using pipx

[jdx]

the precompiled pythons don't work with poetry. You need to reinstall python after running mise settings set python_compile 1. I don't know why they don't work with poetry but the issue is upstream.

[mustafa0x]

an upstream issue needs to be filed, it seems. i couldn't find one.

[egnor]

Note, it's not that precompiled Pythons don't work with poetry, it's that precompiled linux-musl Pythons don't work with poetry. As mentioned in https://github.com/indygreg/python-build-standalone/issues/261 there's some mystery where sometimes mise installs the linux-musl Python and sometimes installs the linux-gnu Python.

On this particular laptop it chose linux-gnu and thus I can install poetry with a prebuilt Python just fine:

% mkdir test
% cd test
% mise u [email protected]
mise installing precompiled python from indygreg/python-build-standalone
mise if you experience issues with this python (e.g.: running poetry), switch to python-build
mise by running: mise settings set python_compile 1
mise [email protected] ✓ installed
mise ~/test/.mise.toml tools: [email protected]
mise [email protected] [email protected]                                                                                                                               
% mise u poetry
mise plugin:poetry ✓ https://github.com/mise-plugins/mise-poetry.git#431c335                                                                                   mise [email protected] ✓ installed
mise ~/test/.mise.toml tools: [email protected]
mise [email protected] [email protected] [email protected]
% which poetry
/home/egnor/.local/share/mise/installs/poetry/latest/bin/poetry
% poetry --version
Poetry (version 1.8.3)

[charliermarsh]

Yeah, just to clarify: the musl builds have a bunch of limitations (see: https://github.com/indygreg/python-build-standalone/issues/86) -- for example, you can't load shard library-based extension modules.

It would be nice if we could avoid this specific issue in python-build-standalone (i.e., pip failing on a dlopen call), but you generally want to be using the non-musl builds. If the musl builds are being installed, there may be an issue here rather than in python-build-standalone?

[jdx]

frankly I have no idea what it does, but this change does appear to have fixed this issue for users using mise-poetry: https://github.com/mise-plugins/mise-poetry/commit/431c335b47be548500c50bd631e30a66b4370788

of course if you just pip install poetry that won't get used though

[charliermarsh]

Interesting... I submitted a PR to pip that I think would avoid this specific failure: https://github.com/pypa/pip/pull/12716. But not sure if it will be accepted or on what timeline it would be released. It should resolve the issue from the python-build-standalone side though, I think.