`poetry run` can fail for scripts defined in `[project.scripts]` if PATH is too long
Description
On Windows, defining a script under [project.scripts] (importantly, not under [tool.poetry.scripts]), and invoking it with poetry run myscript while the PATH environment variable is near or over ~8192 characters, results in:
'myscript' is not recognized as an internal or external command,
operable program or batch file.
This is the bug described in https://github.com/python/cpython/issues/137254, caused by the following code: https://github.com/python-poetry/poetry/blob/b580e8aa4fbce53569420e7b42568dfd9e73519f/src/poetry/utils/env/base_env.py#L445-L446
Notes
While inspecting the Poetry source code, I noticed a few things.
Scripts defined in [tool.poetry.scripts] get special treatment, where they also go through base_env.execute, but the call is made with python -c <inline script> based on the entrypoint's definition, rather than actually calling myscript.cmd (see RunCommand.handle(), RunCommand.run_script()). Scripts defined in [project.scripts] are not taken into account when the check is made, so they fallback to the regular invocation logic.
In Env.get_command_from_bin(), there is some Windows-specific logic that attempts to find <name>.exe in the venv's Scripts directory and resolve its full path. This approach would work. However, the entrypoint is actually <name>.cmd, and so it dodges this particular check.
I think the most reliable approach to resolve this issue would be to use shutil.which(bin, path=env.get("PATH")) in some capacity, as mentioned in the CPython issue. This ensures the script's full path is correctly resolved based on the virtual environment's path, before the call to subprocess.Popen, rather than delegating that logic to the shell. This approach might also allow to remove (or at least trim down) the logic in Env.get_command_from_bin() and Env._bin().
Reproducing
The difficulty in reproducing this issue is getting the PATH to be this long in the first place, as it cannot feasibly be done from cmd.exe, due to its own command length limitations. To work around that, here is a script which creates a ridiculously long path and invokes poetry run ... for the scripts as defined in the pyproject.toml below.
# do_poetry_run.py
import os, subprocess, shutil
fake_path = "C:\\NotAPath"
env = os.environ.copy()
path_elems = [env["PATH"]]
for _ in range(1000):
path_elems.append(fake_path)
env["PATH"] = os.pathsep.join(path_elems)
print("PATH length:", len(env["PATH"]))
poetry = shutil.which("poetry") or "poetry"
try:
subprocess.run([poetry, "run", "pepscript"], env=env)
except Exception as e:
print(f"Error running pepscript: {e}")
try:
subprocess.run([poetry, "run", "poetryscript"], env=env)
except Exception as e:
print(f"Error running poetryscript: {e}")
Workarounds
- Define the script in
tool.poetry.scriptsrather thanproject.scripts - Don't have an absurdly long PATH :)
Poetry Installation Method
pipx
Operating System
Windows 11
Poetry Version
2.1.3
Poetry Configuration
cache-dir = "C:\\Users\\martin.boisvert\\AppData\\Local\\pypoetry\\Cache"
data-dir = "C:\\Users\\martin.boisvert\\AppData\\Roaming\\pypoetry"
installer.max-workers = null
installer.no-binary = null
installer.only-binary = null
installer.parallel = true
installer.re-resolve = true
keyring.enabled = true
python.installation-dir = "{data-dir}\\python" # C:\Users\martin.boisvert\AppData\Roaming\pypoetry\python
requests.max-retries = 0
solver.lazy-wheel = true
system-git-client = false
virtualenvs.create = true
virtualenvs.in-project = null
virtualenvs.options.always-copy = false
virtualenvs.options.no-pip = false
virtualenvs.options.system-site-packages = false
virtualenvs.path = "{cache-dir}\\virtualenvs" # C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs
virtualenvs.prompt = "{project_name}-py{python_version}"
virtualenvs.use-poetry-python = false
Python Sysconfig
sysconfig.log
Platform: "win-amd64"
Python version: "3.13"
Current installation scheme: "venv"
Paths:
data = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13"
include = "C:\Python313\Include"
platinclude = "C:\Python313\Include"
platlib = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Lib\site-packages"
platstdlib = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Lib"
purelib = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Lib\site-packages"
scripts = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Scripts"
stdlib = "C:\Python313\Lib"
Variables:
BINDIR = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Scripts"
BINLIBDEST = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13\Lib"
EXE = ".exe"
EXT_SUFFIX = ".cp313-win_amd64.pyd"
INCLUDEPY = "C:\Python313\Include"
LDLIBRARY = "python313.dll"
LIBDEST = "C:\Python313\Lib"
LIBDIR = "C:\Python313\libs"
LIBRARY = "python313.dll"
Py_GIL_DISABLED = "0"
SOABI = "cp313-win_amd64"
TZPATH = ""
VERSION = "313"
VPATH = "..\.."
abi_thread = ""
abiflags = ""
base = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13"
exec_prefix = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13"
implementation = "Python"
implementation_lower = "python"
installed_base = "C:\Python313"
installed_platbase = "C:\Python313"
platbase = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13"
platlibdir = "DLLs"
prefix = "C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\poetry-gui-scripts-GiBywCFm-py3.13"
projectbase = "C:\Python313"
py_version = "3.13.3"
py_version_nodot = "313"
py_version_nodot_plat = "313"
py_version_short = "3.13"
srcdir = "C:\Python313"
userbase = "C:\Users\martin.boisvert\AppData\Roaming\Python"
Example pyproject.toml
[project]
name = "py-env-truncation"
version = "0.1.0"
description = ""
requires-python = ">=3.13"
[project.scripts]
pepscript = "py_env_truncation.__main__:main"
[tool.poetry.scripts]
poetryscript = "py_env_truncation.__main__:main"
[build-system]
requires = ["poetry-core>=2.0.0,<3.0.0"]
build-backend = "poetry.core.masonry.api"
Poetry Runtime Logs
poetry-runtime.log
C:\Work\Demos\py-env-truncation
(py-env-truncation-py3.13) λ do_poetry_run.py
PATH length: 14940
Loading configuration file C:\Users\martin.boisvert\AppData\Roaming\pypoetry\config.toml
Loading configuration file C:\Users\martin.boisvert\AppData\Roaming\pypoetry\auth.toml
Using virtualenv: C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\py-env-truncation-dNVbun6Y-py3.13
'pepscript' is not recognized as an internal or external command,
operable program or batch file.
Loading configuration file C:\Users\martin.boisvert\AppData\Roaming\pypoetry\config.toml
Loading configuration file C:\Users\martin.boisvert\AppData\Roaming\pypoetry\auth.toml
Using virtualenv: C:\Users\martin.boisvert\AppData\Local\pypoetry\Cache\virtualenvs\py-env-truncation-dNVbun6Y-py3.13
Warning: 'poetryscript' is an entry point defined in pyproject.toml, but it's not installed as a script. You may get improper `sys.argv[0]`.
The support to run uninstalled scripts will be removed in a future release.
Run `poetry install` to resolve and get rid of this message.
Hello, world!
The difficulty in reproducing this issue is getting the PATH to be this long in the first place, as it cannot feasibly be done from cmd.exe
I expect this will therefore get the attention that it deserves...! But if this is for some reason important to you then I imagine a pull request is welcome.
I expect this will therefore get the attention that it deserves...!
😅 Fair enough! I'll see if I can put a bit of time into this issue tomorrow.
(For context, In my situation, poetry run is being invoked from a C++ dev environment where a bunch of separate DLL paths are added to the PATH to allow the script to find locally-built .pyd extension modules, as well as those modules' DLL dependencies. It's certainly not a common situation.)