Poetry cannot be installed on Github Actions runner using latest installer and Python 3.8.

[KelSolaar]

• [x] I am on the latest Poetry version.
• [x] I have searched the issues of this repo and believe that this is not a duplicate.
• [N/A] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• OS version and name: Windows
• Poetry version: Latest
• Link of a Gist with the contents of your pyproject.toml file: N/A

Issue

After changing from the old installer process, i.e. curl -L https://raw.githubusercontent.com/sdispater/poetry/master/get-poetry.py -o get-poetry.py to the new installer process, i.e. curl -sSL https://install.python-poetry.org/ | POETRY_HOME=$HOME/.poetry python3 -, my builds are failing on Github Actions and Python 3.8: Poetry cannot be installed:

022-09-17T01:37:56.7561313Z ##[group]Run curl -sSL https://install.python-poetry.org | python3 -
2022-09-17T01:37:56.7561741Z [36;1mcurl -sSL https://install.python-poetry.org | python3 -[0m
2022-09-17T01:37:56.7572615Z shell: C:\Program Files\Git\bin\bash.EXE --noprofile --norc -e -o pipefail {0}
2022-09-17T01:37:56.7572971Z env:
2022-09-17T01:37:56.7573137Z   CI_PYTHON_VERSION: 3.8
2022-09-17T01:37:56.7573406Z   pythonLocation: C:\hostedtoolcache\windows\Python\3.8.10\x64
2022-09-17T01:37:56.7573650Z ##[endgroup]
2022-09-17T01:37:59.4702925Z Retrieving Poetry metadata
2022-09-17T01:37:59.4703126Z 
2022-09-17T01:37:59.4703425Z # Welcome to Poetry!
2022-09-17T01:37:59.4703691Z 
2022-09-17T01:37:59.4703874Z This will download and install the latest version of Poetry,
2022-09-17T01:37:59.4704250Z a dependency and package manager for Python.
2022-09-17T01:37:59.4704413Z 
2022-09-17T01:37:59.4704801Z It will add the `poetry` command to Poetry's bin directory, located at:
2022-09-17T01:37:59.4705053Z 
2022-09-17T01:37:59.4705200Z C:\Users\runneradmin\AppData\Roaming\Python\Scripts
2022-09-17T01:37:59.4705368Z 
2022-09-17T01:37:59.4705644Z You can uninstall at any time by executing this script with the --uninstall option,
2022-09-17T01:37:59.4705956Z and these changes will be reverted.
2022-09-17T01:37:59.4706100Z 
2022-09-17T01:37:59.4706188Z Installing Poetry (1.2.1)
2022-09-17T01:37:59.4706429Z Installing Poetry (1.2.1): Creating environment
2022-09-17T01:37:59.4706740Z Installing Poetry (1.2.1): An error occurred. Removing partial environment.
2022-09-17T01:37:59.4963567Z Traceback (most recent call last):
2022-09-17T01:37:59.4970320Z   File "<stdin>", line 935, in <module>
2022-09-17T01:37:59.4970900Z   File "<stdin>", line 914, in main
2022-09-17T01:37:59.4971289Z   File "<stdin>", line 545, in run
2022-09-17T01:37:59.4971657Z   File "<stdin>", line 566, in install
2022-09-17T01:37:59.4972175Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\contextlib.py", line 113, in __enter__
2022-09-17T01:37:59.4972602Z     return next(self.gen)
2022-09-17T01:37:59.4972971Z   File "<stdin>", line 638, in make_env
2022-09-17T01:37:59.4973331Z   File "<stdin>", line 624, in make_env
2022-09-17T01:37:59.4973679Z   File "<stdin>", line 314, in make
2022-09-17T01:37:59.4974144Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 68, in create
2022-09-17T01:37:59.4974571Z     self._setup_pip(context)
2022-09-17T01:37:59.4975025Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\venv\__init__.py", line 289, in _setup_pip
2022-09-17T01:37:59.4975508Z     subprocess.check_output(cmd, stderr=subprocess.STDOUT)
2022-09-17T01:37:59.4976022Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 415, in check_output
2022-09-17T01:37:59.4976513Z     return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
2022-09-17T01:37:59.4977012Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 493, in run
2022-09-17T01:37:59.4977453Z     with Popen(*popenargs, **kwargs) as process:
2022-09-17T01:37:59.4977935Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 858, in __init__
2022-09-17T01:37:59.4978405Z     self._execute_child(args, executable, preexec_fn, close_fds,
2022-09-17T01:37:59.4978916Z   File "C:\hostedtoolcache\windows\Python\3.8.10\x64\lib\subprocess.py", line 1311, in _execute_child
2022-09-17T01:37:59.4979397Z     hp, ht, pid, tid = _winapi.CreateProcess(executable, args,
2022-09-17T01:37:59.4979834Z FileNotFoundError: [WinError 2] The system cannot find the file specified
2022-09-17T01:37:59.5142470Z ##[error]Process completed with exit code 1.
2022-09-17T01:37:59.5272597Z Cleaning up orphan processes

An exception is raised in the new installer make definition:

    @classmethod
    def make(cls, target: Path) -> "VirtualEnvironment":
        try:
            # on some linux distributions (eg: debian), the distribution provided python
            # installation might not include ensurepip, causing the venv module to
            # fail when attempting to create a virtual environment
            # we import ensurepip but do not use it explicitly here
            import ensurepip  # noqa: F401
            import venv

            builder = venv.EnvBuilder(clear=True, with_pip=True, symlinks=False)
            context = builder.ensure_directories(target)

            if (
                WINDOWS
                and hasattr(context, "env_exec_cmd")
                and context.env_exe != context.env_exec_cmd
            ):
                target = target.resolve()

            builder.create(target)
        except ImportError:
            # fallback to using virtualenv package if venv is not available, eg: ubuntu
            python_version = f"{sys.version_info.major}.{sys.version_info.minor}"
            virtualenv_bootstrap_url = (
                f"https://bootstrap.pypa.io/virtualenv/{python_version}/virtualenv.pyz"
            )

            with tempfile.TemporaryDirectory(prefix="poetry-installer") as temp_dir:
                virtualenv_pyz = Path(temp_dir) / "virtualenv.pyz"
                request = Request(
                    virtualenv_bootstrap_url, headers={"User-Agent": "Python Poetry"}
                )
                virtualenv_pyz.write_bytes(urlopen(request).read())
                cls.run(
                    sys.executable, virtualenv_pyz, "--clear", "--always-copy", target
                )

        # We add a special file so that Poetry can detect
        # its own virtual environment
        target.joinpath("poetry_env").touch()

        env = cls(target)

        # we do this here to ensure that outdated system default pip does not trigger older bugs
        env.pip("install", "--disable-pip-version-check", "--upgrade", "pip")

        return env

Specifically, the venv._setup_pip definition seems to not find the Python interpreter for the Poetry environment:

    def _setup_pip(self, context):
        """Installs or upgrades pip in a virtual environment"""
        # We run ensurepip in isolated mode to avoid side effects from
        # environment vars, the current directory and anything else
        # intended for the global Python environment
        cmd = [context.env_exec_cmd, '-Im', 'ensurepip', '--upgrade',
                                                         '--default-pip']
        subprocess.check_output(cmd, stderr=subprocess.STDOUT)

I have a repo with a small repro case here: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3071692472/jobs/4962601087

[neersighted]

This looks like an interesting edge case with Python 3.8 and Git Bash (or maybe MSYS bash in general?) triggering a bug in venv. See our CI for 3.8 here not using a pipe. Powershell also works very well (and is what is officially documented):

https://github.com/python-poetry/install.python-poetry.org/actions/runs/3005610039/jobs/4826098405

I suppose we could probably work around this by using the virtualenv zipapp in this case, but I wonder if it's worth fixing as we do not officially suggest using anything but Powershell on Windows (CMD and Git Bash are not tested and best effort only as you can always use Powershell). We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

[KelSolaar]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

That sound sensible to me!

[Secrus]

We could just consider it a known issue and let it expire with support for 3.8 in 6 months.

I am pretty sure Python 3.8 will be supported for the next 2 years (source). It's Python 3.7 that is EOL in 9 months (not 6).

[KelSolaar]

From a pure selfish standpoint and because I tend to follow Numpy schedule, it is April 2023:

On Apr 14, 2023 drop support for Python 3.8 (initially released on Oct 14, 2019)

[neersighted]

Ah, looks like I got the Numpy schedule in my head. @KelSolaar, could you try reproducing with Python 3.7 just for giggles?

[KelSolaar]

I just did: https://github.com/KelSolaar/ActionsRamblings/actions/runs/3108121099/jobs/5036970221

[neersighted]

Good to know it's the same on 3.7 -- we'll probably just declare this a known issue with Python 3.7 & 3.8 + MSYS Bash.

[Secrus]

So, the issue is only present when using the bash shell on Windows runner?

[neersighted]

That is correct, yes. We haven't drilled down deep enough to figure out if it's truly MSYS bash in general, or just the version shipped by Git (bash on GHA actions is Git Bash), but I think that's mostly unnecessary.

[sondrelg]

Hi 👋 I maintain a Github action for setting up Poetry (this one), and we've had a few reports of broken 3.8 workflows after switching to this installer.

I'm fine with this not being fixed, but wanted to ask: would you recommend using the old script from the main Poetry repo for these targets? That's what I'm planning to do, unless you have a better idea? 🙂

[KelSolaar]

I would need to double-check but I think the main reason for me to open this issue is that the old installer was not working, I updated to the new one which then broke with Python 3.8.

[sondrelg]

The action has previously used an older version (https://raw.githubusercontent.com/python-poetry/poetry/48339106eb0d403a3c66519317488c8185844b32/install-poetry.py) which works fine 🙂

[Secrus]

Hi wave I maintain a Github action for setting up Poetry (this one), and we've had a few reports of broken 3.8 workflows after switching to this installer.

I'm fine with this not being fixed, but wanted to ask: would you recommend using the old script from the main Poetry repo for these targets? That's what I'm planning to do, unless you have a better idea? slightly_smiling_face

In theory, it will install Poetry just fine, however, that script is no longer maintained and any changes to the installer are made here, in this repository. The script in Poetry main repo is scheduled for removal python-poetry/poetry#6676

[neersighted]

We should work just fine with Powershell -- so please use that. If you are experiencing breakage with Python 3.8 and Powershell, that is a different issue from what is tracked here and worth reporting. I would not depend on the old script in the Poetry repo.

[neersighted]

Also, if you really have to use the installer in Git Bash, and the old script works, it's probably worth analyzing why. The original theory here was that it was mostly external to the installer -- but given the old script works, that's evidently incorrect and it's worth figuring out why.

[sondrelg]

From what I can gather, this is the failing call in the installation script, and it fails because Popen is called with 'C:\\Users\\runneradmin\\.local\\venv\\Scripts\\python3.exe' (in my test where we force .local as the install path) when the contents of that dir are ['python.exe', 'pythonw.exe'].

This is a bit odd since the python3.exe executable name is defined here using sys. _base_executable, before the error reported above is raised downstream from here.

I don't have a Windows machine available, so not sure I'm the best to create a fix for this, but the problem seems simple at least 🙂 Any thoughts on this?

[miigotu]

I imagine if python3.exe was added here like python3 is added in the != nt case, it would solve the issue?

[sondrelg]

The old script we were using just did python = env_path.joinpath("Scripts/python.exe") (ref)

[neersighted]

Looks like a bit of misdiagnosis -- the issues are in the internals of the venv module and not related to how we invoke pip. It appears the old script unconditionally uses virtualenv, and the venv API is just broken on some older Python versions under git bash. Our immediate choices (for a quick fix) are to either use venv via the CLI entrypoint/interface (and not the Python API), or to use virtualenv on some (or all) platforms. As we're trying to avoid the external dependency, I think we just need to special case and use virtualenv on these broken Python-windows combinations.

[miigotu]

virtualenv doesn't seem like the best choice, since venv is the standard way since 3.6 and all the older versions are way past eol.

It frustrates me when poetry uses virtualenv and pycharm tells me it's a broken env and repairs it, and then poetry tells me it's broken and repairs it lol.

Especially because ubuntu 22.04 built in python3.10 creates venv/local/bin instead of venv/bin so the poetry support built into pycharm doesn't work, it compounds the issue.

[neersighted]

You're mixing up a lot of issues here, @miigotu. virtualenv is used by Poetry proper, and creates valid virtual environments according to PEP 405. If PyCharm does not recognize them as valid, that is PyCharm's fault.

The issue with Ubuntu 22.04 is a bug in Ubuntu itself, and not virtualenv's fault. There are multiple upstream bugs open for it.

If the stdlib venv module is broken under Windows on these Python versions, I don't see a choice other than using the virtualenv module as a zipapp. We already do this on platforms where venv is unavailable (Ubuntu distro Pythons without the -venv package).

[miigotu]

You're mixing up a lot of issues here, @miigotu. virtualenv is used by Poetry proper, and creates valid virtual environments according to PEP 405. If PyCharm does not recognize them as valid, that is PyCharm's fault.

The issue with Ubuntu 22.04 is a bug in Ubuntu itself, and not virtualenv's fault. There are multiple upstream bugs open for it.

If the stdlib venv module is broken under Windows on these Python versions, I don't see a choice other than using the virtualenv module as a zipapp. We already do this on platforms where venv is unavailable (Ubuntu distro Pythons without the -venv package).

Not mixing up anything at all. The issue with windows runners is how the virtual environment is created currently.

The question is, according to your last comment, either the venv invocation needs fixed or use virtualenv on some older platforms. I'm giving reasons why using virtualenv on anything past python3.6 is anti-progressive. Those reasons are undoubtedly going to touch unrelated subjects when you are choosing a route to take with a patchy fix.

After 30 years as a developer I tend to take all information into account when making design choices.

[neersighted]

This is the repository for the installer script, not for Poetry. This script has no influence on how we create virtual environments in Poetry. If you think we should prefer venv, please open an issue on the correct repo.

The only thing virtual env creation code in this script does is create the environment Poetry and its dependencies are installed into. If venv is broken for the installer's usage on certain versions of Python, falling back to virtualenv is the sane thing to do.

[miigotu]

If it's not the installer that is causing the issue that when it calls the venv code, why doesn't it happen when you create a venv using the interpreter directly on windows?

Deflection isn't a solution. And adding a fallback to a previous standard instead of fixing the issue directly just ends up making tools that depend on the installer not work for longer. If it takes 6 months for a mainstream tool to make the changes (and haven't yet) to work with the latest version and then you add a different behavior in some instances, how long do you think it will be before those mainstream products work with your new changes? Usually about a year, or longer.

https://stackoverflow.com/questions/41573587/what-is-the-difference-between-venv-pyvenv-pyenv-virtualenv-virtualenvwrappe#comment74964971_41573588

https://stackoverflow.com/a/47559925

I'll just keep my comments out of it. The answer shouldn't be everything isn't our problem and splitting the repo into 3,4,5 repos just so you can point fingers to somewhere else. The answer also shouldn't be well we make it however and everyone just has to play catch-up, when in effect it's moving backwards. virtualenv is deprecated for a reason.

[neersighted]

virtualenv is actively developed and is more fully featured than the minimal implementation in the standard library. Please stop driving this issue off topic -- this issue is for a problem creating a virtual environment to install Poetry into, using the stdlib venv module with with_pip=True.

The previous iteration of this installer works because it doesn't rely on the presence of the stdlib module. While I would like to get to the bottom of that, the immediate fix to ensure users can continue to rely on this installer is to prefer virtualenv on these older Pythons. Then we can figure out what the bug in the stdlib module is, and if we can work around it (e.g. use get-pip.py instead of relying on the module to call ensurepip for us).

[miigotu]

That's not the problem at all. The problem is the script is called with a hashbang of python3, and in the venv there is no python3, just python (which is python 3). You call that subprocess with sys.cmd_exe, generated from sys.executable.

Python has no issues creating venv on python3.3+, and virtualenv IS deprecated in python3.8+

Just say you don't know what the issue is and how to fix it, or that you don't want to work on it if that's what it is.

[neersighted]

The code in question that generates context.env_exec_cmd/context.env_exec is in the venv module, and there is no standard shebang invocation -- on Windows, we suggest use of the py.exe launcher. That launcher is invoking python3.exe in turn (based on the python3 shebang, however, py -3 would have the same result). When using venv.EnvBuilder, there is no facility to control the Python executable -- it is derived internal to the module from sys.executable.

[miigotu]

Guess it's time to create my own installer. Windows requires no super special handling for venv. Have a good day.

If a user doesn't have python2 installed, and calls the installer with python.exe ... the install passes.

[systematicguy]

Arriving from https://github.com/python-poetry/poetry/issues/3388

I have the same issue when trying to pipe into python3 (worked around for now piping into python.) I am on windows 10, using pyenv-win, my global python is set to 3.9.6.

[stewartmiles]

FYI: I came across this while root causing this issue outside of poetry (related is https://github.com/actions/runner-images/issues/2690). The problem is due to https://github.com/python/cpython/pull/25216 . It's possible to workaround this by subclassing venv.EnvBuilder, and then overloading venv.EnvBuilder.ensure_directories() and venv.EnvBuilder.setup_python() to patch context.env_exe to point at the copied executable filename (i.e one of {'python.exe', 'python_d.exe', 'pythonw.exe', 'pythonw_d.exe'}) instead of the name derived from the current interpreter (sys._base_executable).