devbox
devbox copied to clipboard
jetify-com
[Bug]: poetry self add <plugin> fails in devbox
Current Behavior (bug) I'm getting a ton of Nix errors when trying to add a poetry plugin inside a devbox shell:
justins-MacBook-Pro:trzpyutils justin$ which python
justins-MacBook-Pro:trzpyutils justin$ which poetry
justins-MacBook-Pro:trzpyutils justin$ devbox shell
Info: New devbox available: 0.8.4 -> v0.8.5. Please run `devbox version update`.
Starting a devbox shell...
You can activate the virtual environment by running 'source $VENV_DIR/bin/activate'
Using virtualenv: /Users/justin/git/gl/****/trzpyutils/.venv
(devbox) justin@justins-MacBook-Pro trzpyutils % which poetry
/Users/justin/git/gl/****/trzpyutils/.devbox/nix/profile/default/bin/poetry
(devbox) justin@justins-MacBook-Pro trzpyutils % which python
/Users/justin/git/gl/****/trzpyutils/.devbox/nix/profile/default/bin/python
(devbox) justin@justins-MacBook-Pro trzpyutils % poetry self add "poetry-dynamic-versioning"
Resulting error here:
Using version ^1.2.0 for poetry-dynamic-versioning
Updating dependencies
Resolving dependencies...
Package operations: 5 installs, 37 updates, 0 removals
• Updating certifi (2023.7.22 /nix/store/mn3955icdlwwfn1l7f9wlmg2fzpba48h-python3.11-certifi-2023.07.22/lib/python3.11/site-packages -> 2023.11.17)
• Updating charset-normalizer (3.2.0 /nix/store/748ky268clczsa4iqr5lgzgwz93gnjs4-python3.11-charset-normalizer-3.2.0/lib/python3.11/site-packages -> 3.3.2)
• Updating idna (3.4 /nix/store/npbk2j4vz2ijymi6s8ds8fnzdmnjzgml-python3.11-idna-3.4/lib/python3.11/site-packages -> 3.6)
• Updating more-itertools (10.1.0 /nix/store/gh9qqy866kgrb03pqwrvv0d9ly5h7qhh-python3.11-more-itertools-10.1.0/lib/python3.11/site-packages -> 10.1.0)
• Updating pycparser (2.21 /nix/store/8j6n9d5zqlnic7a94hsiyil29gl1mcps-python3.11-pycparser-2.21/lib/python3.11/site-packages -> 2.21)
• Updating urllib3 (2.0.7 /nix/store/jb519vkvay5kcpki6sbgr247f80pci7l-python3.11-urllib3-2.0.7/lib/python3.11/site-packages -> 2.1.0)
• Updating zipp (3.16.2 /nix/store/v28s3qh0absn3wrgg8d4zmlljqd97s36-python3.11-zipp-3.16.2/lib/python3.11/site-packages -> 3.17.0)
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'urllib3', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'urllib3', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install urllib3.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'more-itertools', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'more-itertools', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install more-itertools.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'zipp', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'zipp', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install zipp.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'pycparser', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'pycparser', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install pycparser.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'certifi', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'certifi', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install certifi.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'idna', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'idna', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install idna.
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'charset-normalizer', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'charset-normalizer', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
at /nix/store/xdp0dyglkkfrny00ij665wg1jgywy0lj-python3.11-poetry-1.7.1/lib/python3.11/site-packages/poetry/utils/env/base_env.py:354 in _run
350│ output = subprocess.check_output(
351│ cmd, stderr=stderr, env=env, text=True, **kwargs
352│ )
353│ except CalledProcessError as e:
→ 354│ raise EnvCommandError(e)
355│
356│ return output
357│
358│ def execute(self, bin: str, *args: str, **kwargs: Any) -> int:
Cannot install charset-normalizer.
```
Additional context
Please include the output of devbox version -v and
a copy of your devbox.json file.
I can reproduce this. Still investigating, but it looks like PYTHONPATH is still set to the Nix store after activating the virtual environment, so poetry is trying to install dependencies there.
$ devbox shell
(devbox) $ . $VENV_DIR/bin/activate
(.venv) (devbox) $ poetry self add "poetry-dynamic-versioning"
Using version ^1.2.0 for poetry-dynamic-versioning
Updating dependencies
Resolving dependencies... (2.4s)
Package operations: 5 installs, 37 updates, 0 removals
• Updating certifi (2023.7.22 /nix/store/mn3955icdlwwfn1l7f9wlmg2fzpba48h-python3.11-certifi-2023.07.22/lib/python3.11/site-packages -> 2023.11.17): Installing...
• Updating charset-normalizer (3.2.0 /nix/store/748ky268clczsa4iqr5lgzgwz93gnjs4-python3.11-charset-normalizer-3.2.0/lib/python3.11/site-packages -> 3.3.2): Failed
CalledProcessError
Command '['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'charset-normalizer', '-y']' returned non-zero exit status 1.
at /nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/lib/python3.11/subprocess.py:571 in run
567│ # We don't call process.wait() as .__exit__ does that for us.
568│ raise
569│ retcode = process.poll()
570│ if check and retcode:
→ 571│ raise CalledProcessError(retcode, process.args,
572│ output=stdout, stderr=stderr)
573│ return CompletedProcess(process.args, retcode, stdout, stderr)
574│
575│
The following error occurred when trying to handle this error:
EnvCommandError
Command ['/nix/store/z0nnwdh4p692yvf3qgwgj3bn78w99q6y-python3-3.11.6/bin/python3.11', '/nix/store/09ifrdls58b4hqkscmxa6ps7w4pb9wa5-python3.11-virtualenv-20.24.5/lib/python3.11/site-packages/virtualenv/seed/wheels/embed/pip-23.2.1-py3-none-any.whl/pip', 'uninstall', 'charset-normalizer', '-y'] errored with the following return code 1
Output:
error: externally-managed-environment
× This environment is externally managed
╰─> This command has been disabled as it tries to modify the immutable
`/nix/store` filesystem.
To use Python with Nix and nixpkgs, have a look at the online documentation:
<https://nixos.org/manual/nixpkgs/stable/#python>.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
(.venv) (devbox) $ echo $PYTHONPATH
/nix/store/hg98gh5lcan57ysy3viqr2irxsbqv42s-python3-3.13.0a2/lib/python3.13/site-packages
Oh, I just noticed that this is a plugin being added to poetry's own environment, not the virtual environment. I'm trying to use devbox add python311Packages.poetry-dynamic-versioning instead, but poetry doesn't seem to pick it up.
@justinTM I looked into this for a bit, but unfortunately I don't think there's an easy way to add poetry plugins right now. I even tried to get poetry-dynamic-versioning working directly from a Nix flake, but couldn't get it to show up in poetry self show plugins. This issue has some more details on poetry plugins + Nix.
We'll have to look into prioritizing some work + debugging time to get this working smoothly.