github
Exception in configuring venv when the project has a `setup.py` file
Hey!
When playing around with python CodeQL analysis I stumbled across an exception during the init phase:
Installing collected packages: virtualenv-clone, pipenv
Successfully installed pipenv-2022.9.8 virtualenv-clone-0.5.7
+ command -v python2
/usr/bin/python3 -B /home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py /opt/hostedtoolcache/CodeQL/0.0.0-20220825/x64/codeql
Source root appears to be the real root.
Will try to guess Python version, as it was not specified in `lgtm.yml`
Trying to guess Python version based on Trove classifiers in setup.py
Found Trove classifiers for both Python 2 and Python 3 in setup.py -- will use Python 3
Found setup.py, will install package with pip in editable mode
Creating venv in /home/runner/work/python-systemd/codeql-action-python-autoinstall
+ python3 -m virtualenv /home/runner/work/python-systemd/codeql-action-python-autoinstall
/home/runner/.local/lib/python3.10/site-packages/setuptools/command/install.py:34: SetuptoolsDeprecationWarning: setup.py install is deprecated. Use build and pip and other standards-based tools.
warnings.warn(
created virtual environment CPython3.10.4.final.0-64 in 491ms
creator CPython3Posix(dest=/home/runner/work/python-systemd/codeql-action-python-autoinstall, clear=False, no_vcs_ignore=False, global=False)
seeder FromAppData(download=False, pip=bundle, setuptools=bundle, wheel=bundle, via=copy, app_data_dir=/home/runner/.local/share/virtualenv)
added seed packages: pip==21.3.1, setuptools==58.3.0, wheel==0.37.0
activators BashActivator,CShellActivator,FishActivator,NushellActivator,PowerShellActivator,PythonActivator
+ /home/runner/work/python-systemd/codeql-action-python-autoinstall/bin/pip install -e .
Traceback (most recent call last):
File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 175, in <module>
python_executable_path = install_packages(codeql_base_dir)
File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 163, in install_packages
return install_with_setup_py(version)
File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 134, in install_with_setup_py
_check_call([venv_pip, 'install', '-e', '.'])
File "/home/runner/work/_actions/github/codeql-action/v2/python-setup/auto_install_packages.py", line 14, in _check_call
subprocess.check_call(command, stdin=subprocess.DEVNULL)
File "/usr/lib/python3.10/subprocess.py", line 364, in check_call
retcode = call(*popenargs, **kwargs)
File "/usr/lib/python3.10/subprocess.py", line 345, in call
with Popen(*popenargs, **kwargs) as p:
File "/usr/lib/python3.10/subprocess.py", line 966, in __init__
self._execute_child(args, executable, preexec_fn, close_fds,
File "/usr/lib/python3.10/subprocess.py", line 1842, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: '/home/runner/work/python-systemd/codeql-action-python-autoinstall/bin/pip'
Warning: An error occurred while trying to automatically install Python dependencies: Error: The process '/usr/bin/python3' failed with exit code 1
Please make sure any necessary dependencies are installed before calling the codeql-action/analyze step, and add a 'setup-python-dependencies: false' argument to this step to disable our automatic dependency installation and avoid this warning.
Taken from: https://github.com/systemd/python-systemd/actions/runs/3062293076/jobs/4943107761#step:3:439 Also appears in: https://github.com/systemd/mkosi/actions/runs/3051938808/jobs/4920754739#step:3:443
It doesn't seem to be fatal (at least in the projects I currently tested it with), but it generates a warning on the results overview page:
This seems to happen only when the project has a setup.py file; in projects without it the relevant step fails cleanly:
Successfully installed pipenv-2022.9.8 virtualenv-clone-0.5.7
+ command -v python2
/usr/bin/python3 -B /home/runner/work/_actions/github/codeql-action/0c670bbf0414f39666df6ce8e718ec5662c21e03/python-setup/auto_install_packages.py /opt/hostedtoolcache/CodeQL/0.0.0-20220825/x64/codeql
No directories containing root identifiers were found. Returning working directory as root.
Will try to guess Python version, as it was not specified in `lgtm.yml`
Trying to guess Python version based on Trove classifiers in setup.py
Did not find setup.py (expected it to be at /home/runner/work/systemd/systemd/setup.py)
Trying to guess Python version based on travis file
Did not find any travis files (expected them at either ['/home/runner/work/systemd/systemd/.travis.yml', '/home/runner/work/systemd/systemd/travis.yml'])
Trying to guess Python version based on installed versions
Only Python 3 installed -- will use that version
was not able to install packages automatically
See: https://github.com/systemd/systemd/actions/runs/3061829175/jobs/4942084517#step:3:459 or: https://github.com/systemd/systemd-centos-ci/actions/runs/3060692901/jobs/4939517444#step:3:424
Hi @mrc0mmand, that does look like a bug, thanks for reporting it :pray: Let me see if I can come up with a fix later today.
I wasn't able to figure out what was going on, but will look more at it Monday.
To unblock you, you can do the dependency installation manually as described here: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#analyzing-python-dependencies
I wasn't able to figure out what was going on, but will look more at it Monday.
To unblock you, you can do the dependency installation manually as described here: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#analyzing-python-dependencies
Hey, thanks for looking into it and no worries, it doesn't really affect any of the projects I tried it with so far.
Great to hear :+1:
I've found the underlying cause. As described here, when using Ubuntu 22.04 with new enough versions of setuptools (60.0.0+), the virtual environment created with virtualenv will put binaries in <venv-path>/local/bin instead of <venv-path>/bin.
Next release after 20.16.5 of virtualenv will include a fix for this (PR).
I've created a temporary fix for this in https://github.com/github/codeql-action/pull/1257. again, thanks for reporting :muscle: (our automated testing didn't catch this, since it's using ubuntu-latest, and not 22.04)