poetry
poetry copied to clipboard
python-poetry
Option to force Poetry to create a virtual environment, even if a virtual env is active
I am using Poetry from within a conda environment; with Poetry being installed by conda (poetry is present in the environment.yaml file). This might not be ideal but for a specific setup this seems to work well.
When running poetry install, a venv is not created because Poetry detect that a virtual environment (the Conda one) is already active.
I believe that we should be able to force Poetry to create a virtual environment anyway: the Conda environment is more "physical" than virtual in this case, as it replaces the system Python.
The same issue occurs when doing poetry shell from within the Conda environment, after creating the virtual environment with the following workaround: I deactivate the Conda environment, use the full path to Poetry (in the bin directory of the Conda environment) and do poetry install.
This is related to #1724 and to #4050, but in the form of an explicit request to add an option.
What would really be nice is a way to have an active underlying conda environment upon which poetry could create a virtual environment for installing dependencies while maintaining access to the underlying conda environment.
A use-case is a group of students working on a shared computer (like an HPC cluster): I would like to create a fairly full featured shared conda environment (called work here) which students have read access to, but not write. Currently poetry will treat this active conda environment as a virtual environment, but fail because it has no write access.
The behaviour I would like (if possible) is for poetry to be able to install or upgrade any missing dependencies as needed, akin to how pip install --user would, but to install these into a custom virtual environment rather than the users ~/.local.
I know that I could do this by manipulating paths PYTHONUSERBASE etc. but it would be really nices if this could "just work" in a way that consistent with general poetry usage.
Thus, daily work looks like:
conda activate work # Activate read-only conda environment
poetry shell # Creates a new virtual environment with access to `work`
python ... # Works for anything using just `work`
poetry add ... # If something is not in `work` (currently fails because of permissions)
poetry install # Installed into the virtual environment like pip install --user
This allows students to get to work as quickly as possible, allowing us to provide most
of what they need in the work environment, but providing them a way to install other
libraries if needed. Everything new needed is recorded in pyproject.toml, so in
combination with the environment file for the work environment, this satisfies the
basic requirements for reproducibility.
When preparing for release, one would add the minimal set of missing dependencies supplied by work into a clean virtual environment:
poetry env use 3.8
poetry shell # Clean environment without `work`
poetry add ... # Clean up all dependences for release.
...
Now students need to work harder to properly define a clean minimal set of dependencies, but this task does not stand in the way of them getting started.
Any suggestions for alternatives would be appreciated.
Edit: It looks like the a project-specific virtualenvs.options.system-site-packages = true config option as requested in #2937 might solve my issue.
One other feature that would be really nice is if poetry shell could activate whatever environment is needed, including calling conda activate. I am not sure how internally this works, but the following workflow would be very useful:
conda activate my_conda_env
poetry env use python # Tell poetry to use the my_conda_env
poetry shell # This works.
After doing this, it would be very nice if this or something similar could store information about the choice of environment in poetry.toml or something so that running poetry shell from within the project first activates the my_conda_env (until poetry env use or similar is called in the future).
I don't use conda, but I second this feature request anyways.
In trying to debug a failing CI pipeline, it helps a lot if the venv is exclusive to the current build. On traditional Jenkins agents, that can only be assured by creating a venv in the current workspace, no matter which environments exist. (One might say it's essential that the venv is exclusive to each build, always.)
Currently, I have no way (?) to force poetry to not use an existing environment (which I can't delete), which may or may not be the root of my problem.
This is of particular concern since running, say, end-to-end tests with poetry run myscript (with myscript declared in pyproject.toml:tool.poetry.scripts) doesn't work after poetry install --no-root; that is, I will have to pollute my "global" virtual environment with the state of the code of the current build, which leads me to believe that concurrent builds of, say, different branches won't work reliably.
PS: Not sure whether poetry install --no-root && poetry run myscript should be a bug report or feature request.
I am also missing this feature, as I am migrating some code from pipenv to poetry. pipenv has PIPENV_IGNORE_VIRTUALENVS which has exactly the effect that's wished for here.
I described a workaround in this answer:
https://stackoverflow.com/questions/70739858/how-to-create-a-brand-new-virtual-environment-or-duplicate-an-existing-one-in-po
However, it would be nice to have the possibility to define more pyproject.toml and create other environments with the framework.
This makes sense. I've had this happen to me again today
pipenv(1) does this behavior
IMO, In this case this would still be valid poetry usage, even if poetry wasn't necessarily installing the packages.
Any more updates/traction on this?
My usecase is upgrading some projects from py3.7 to py3.10 and it's really annoying having two conda envs for each project. It will greatly help forcing poetry to create a local env every time.
Deleting the existing virtualenv directory did help me.
$ poetry run python -m site
sys.path = [
...
'/Users/user-name/Library/Caches/pypoetry/virtualenvs/some-project/lib/python3.10/site-packages',
]
After this I just simply ran
poetry install
and a new virtualenv env was automatically created.
I also added these lines to the project's poetry.toml (even though these are my global configs as well)
[virtualenvs]
create = true
in-project = true
+1 for the conda case, it's pretty problematic in our remote environment, where python interpreter is provided by conda and poetry install doesn't create project-specific venvs...
+1 for the
condacase, it's pretty problematic in our remote environment, wherepythoninterpreter is provided bycondaandpoetry installdoesn't create project-specific venvs...
I hava found PDM, which meets my requirements.
I have found PDM, which meets my requirements.
Wow, PDM is like npm for Python... thanks for sharing!
I faced this problem as well and created a Poetry plugin that fixes this issue for the Conda use case. Although not ideal, this solution seems to work well when I tested it.
Would Poetry maintainers be open to adding a similar config such as virtualenvs.ignore-conda-env to Poetry? I'd be happy to contribute with a PR.
Referencing an earlier comment from @Secrus: https://github.com/python-poetry/poetry/issues/7878#issuecomment-1535496388.
...My guess would be that you have Poetry installed in the same environment as your project, which is discouraged and prone to breakage when your dependencies clash with Poetry's.
This current behavior seems very much in contradiction to the linked guidance. If installing poetry in the same environment is to be discouraged, then attempts to initialize/install a project co-located with poetry should at the very least raise a warning that this is problematic.
I faced this problem as well and created a Poetry plugin that fixes this issue for the Conda use case
poetry-conda has fixed the issue I had with this. Poetry was installed on a (non-virtual) python, but I had a conda environment enabled that didn't contain python for some binary tooling (specifically h5ls and othe inspection tools). Every time trying to e.g. poetry run morgul-prepare it would say
The virtual environment found in /path/to/activated/conda/ENV seems to be broken.
Recreating virtualenv morgul-zttW3Bd7-py3.11 in /poetry_cache_path/virtualenvs/morgul-zttW3Bd7-py3.11
Warning: 'morgul-prepare' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.
The support to run uninstalled scripts will be removed in a future release.
Run `poetry install` to resolve and get rid of this message.
Traceback (most recent call last):
File "<string>", line 1, in <module>
File "/dls/science/users/mep23677/modules/python/3.11.3/lib/python3.11/importlib/__init__.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
File "<frozen importlib._bootstrap_external>", line 940, in exec_module
File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
File "/dls/science/users/mep23677/jungfrau/jungfrau-commissioning/morgul/morgul_prepare.py", line 7, in <module>
import h5py
ModuleNotFoundError: No module named 'h5py'
Then run poetry install, which appears to succeed, then the poetry run .... goes back to the above error message.
Poetry definitely shouldn't be trying to use an existing conda environment that doesn't contain python, and I never want it to, so a systemwide setting that lets's me turn off poetry trying to do this would be perfect.
This is a national lab, so we don't generally have free reign to install system-level packages, and using up-to-date things are usually a mixture of environment modules or conda environments.
The behavior that poetry always prefers an acitvated venv is harmful and leads to stupid workarounds and nonsensical behavior. Normally also normal users have virtual envs on their machine which act as a basis for lets say python 3.10 which is active in the shell.
We can only force creating a venv by doing:
(unset VIRTUAL_ENV && poetry -v install)
which will then read the poetry.toml which has virtualenvs.create=true which then installs into ./.venv and not use the already activated one.
The above workaround however leads to issues with other commands because its not synchronized:
poetry env info --path # might be used in CI scripts to activate the correct venv ... etc etc
will in the same shell not give you the just created , nameley ./.venv but the system venv.
I am not even get the option POETRY_VIRTUALENVS_CREATE in that sense. If you specify true it should always create one, full stop. Because the other behavior "it always prefers already activated venvs" leads to really peculiar side effects in CI and on your local machine.
Please fix this in making this easy and comprehensible.
Encountered this issue as well - would be great to have either:
- Some opt-in/opt-out flag OR
- Prompt when running
poetry initorpoetry install. This way users aren't surprised. e.g.It seems like you have an existing virtualenv active (/path/to/venv). Would you like to use this virtualenv or create a new one?
Without it, you can accidentally install things in a virtual environment you didn't expect to.
I just tried using poetry for the first time following the introduction guide in the documentation. The official documentation is very misleading and contradicting itself, which has brought me here!
In the very first chapter you see this big, red warning:
If you heed this warning, installing poetry in its own environment, and continue with the example you will run into the issue described here in the very next chapter. Poetry will happily use its own environment for the example project without any question or warning.
Using your virtual environment
By default, Poetry creates a virtual environment in {cache-dir}/virtualenvs.
This is just plain wrong at this point. Poetry will not do this if you follow the guide. It will just silently use its own environment.
You can change the cache-dir value by editing the Poetry configuration.
I have not tried this.
Additionally, you can use the virtualenvs.in-project configuration variable to create virtual environments within your project directory.
I have not tried this either, but from what I can see in the related issues, this is also wrong.
At this point I started googling why I couldn't find any virtualenvs created by poetry.
Only after that, does the documentation mention that poetry will prefer an already active virtualenv. This might clear things up if you already have some experience or you have googled the issue. But for a first time user this guide is misleading and unusable.
@maflAT I understand the confusion if you only read the red box. However, if you read further it does not matter which method you choose (pipx, official installer, manual): all methods make sure that poetry's own virtual environment is not activated. Nevertheless, a PR improving the docs (maybe an additional warning that poetry's own environment should not be activated) is always welcome.
I think the problem does not surface when poetry is installed but at a later point in time when used. So I agree, all the methods mentioned here do not activate the environment. But when you actually make use of poetry, say, to run poetry install then you shouldn't do that from poetry's virtual environment although it's really tempting to. And actually, the fact the commands mention poetry install only could let you think you have activated the virtual environment in which you installed poetry (I actually thought so and was really puzzled by poetry install not creating the .venv in my sub-directory although virtualenvs.in-project was set to true.
I tried to update the warning sign in #8833
With the arrival of Pixi, this has become more of a problem. Pixi does not have a base environment by default. If you create a base environment in which to install Python and Poetry, you need to know to call it "base" or else Poetry will not work.
I would recommend this as the solution: Option called virtualenvs.ignore-conda-envs. If true, Poetry behaves as if it does not see the conda environment at all. If false, Poetry always installs into a conda environment when inside one. If a list of strings, Poetry ignores only those environments. The default is ["base"] keeping the current behavior unchanged.
I just came across this issue upon installing a development environment based only on asdf, pipx, and poetry on a machine that had conda used on it previously for other projects.
I set the setting:
poetry config virtualenvs.in-project true
immediately after installing Poetry, so that no matter what, the Poetry usage would be confined to this new repository using this setup.
However, after running poetry install, and to my surprise, no .venv folder was created in the project root directory. It wasn't even created in the poetry config virtualenvs.path directory. Instead, it went and updated an existing conda virtual environment, and it did so without any confirmation. The only way I figured it out was running poetry env info --path. So now Poetry has installed and upgraded the very environment I didn't want it to touch.
To me, this is a bug and not a feature request. Poetry didn't do anything it was supposed to do as specified in its configuration, and it instead did the very thing I did not want it to do with no confirmation prior to doing so.
