salt
salt copied to clipboard
saltstack
[BUG] salt-pip install M2Crypto or libvirt-python failed. Minion-3006.7
Description Using salt-pip to install M2Crypto or libvirt-python failed.
Setup (Please provide relevant configs and/or SLS files (be sure to remove sensitive info. There is no general set-up of Salt.)
Please be as specific as possible and give set-up details.
- [X] on-prem machine
- [X] VM (Virtualbox, KVM, etc. please specify)
Steps to Reproduce the behavior
salt-pip install M2Crypto
Collecting M2Crypto
Using cached M2Crypto-0.41.0.tar.gz (1.2 MB)
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
Error in sitecustomize; set PYTHONVERBOSE for traceback:
AssertionError:
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
main()
File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
execsitecustomize()
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 959, in wrapper
import sitecustomize
File "/tmp/pip-build-env-_5mtlr_1/site/sitecustomize.py", line 22, in <module>
assert not path in sys.path
AssertionError
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
[notice] A new release of pip is available: 23.3.2 -> 24.0
Expected behavior A clear and concise description of what you expected to happen.
Versions Report
salt --versions-report
(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)salt-minion -V
Salt Version:
Salt: 3006.7
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.14.6
cherrypy: 18.6.1
dateutil: 2.8.1
docker-py: Not Installed
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.0.2
M2Crypto: 0.40.1 <--Do not be confused, this is after install of M2crypt
Mako: Not Installed
msgpack: 1.0.2
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 22.0
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.4.8
PyYAML: 6.0.1
PyZMQ: 23.2.0
relenv: 0.15.1
smmap: Not Installed
timelib: 0.2.4
Tornado: 4.5.3
ZMQ: 4.3.4
System Versions:
dist: ubuntu 22.04.2 jammy
locale: utf-8
machine: x86_64
release: 5.15.0-60-generic
system: Linux
version: EdgAIR OS 22.04.2 jammy
Additional context The issue has been resolved, when I create new python virtual env independent from salt and install libvirt-python there. After that the installation was OK. But I was not able to install M2crypt till I install it in independent environment and continue with salt-pip install. IMHO it could have some relation to wheel usage or something.
When running more verbose salt-pip install it provides more information:
/opt/saltstack/salt/salt-pip install libvirt-python -v
#This will provide more inside about the problem
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /opt/saltstack/salt/bin/python3.10 /opt/saltstack/salt/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpkuxa3lja
cwd: /tmp/pip-install-4ur6lpg6/libvirt-python_e5271fd61c4b4477af1b225c9ad0b5b9
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
salt-pip list
Package Version
------------------ ---------
certifi 2023.7.22
cffi 1.14.6
charset-normalizer 3.2.0
cheroot 8.5.2
CherryPy 18.6.1
contextvars 2.4
cryptography 42.0.3
distro 1.5.0
filelock 3.13.3
idna 3.2
immutables 0.15
importlib-metadata 6.0.0
jaraco.classes 3.2.1
jaraco.collections 3.4.0
jaraco.functools 2.0
jaraco.text 3.5.1
Jinja2 3.1.3
jmespath 1.0.1
libvirt-python 10.2.0
looseversion 1.0.2
M2Crypto 0.41.0
MarkupSafe 2.1.2
more-itertools 5.0.0
msgpack 1.0.2
packaging 22.0
pip 23.3.2
portend 2.4
psutil 5.8.0
pycparser 2.21
pycryptodomex 3.19.1
pyOpenSSL 24.0.0
python-dateutil 2.8.1
python-gnupg 0.4.8
pytz 2022.1
PyYAML 6.0.1
pyzmq 23.2.0
relenv 0.15.1
requests 2.31.0
rpm-vercmp 0.1.2
salt 3006.7
setproctitle 1.3.2
setuptools 68.2.2
six 1.16.0
tempora 4.1.1
timelib 0.2.5
urllib3 1.26.18
wheel 0.42.0
xmltodict 0.13.0
zc.lockfile 1.4
zipp 3.6.0
Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey. Please be sure to review our Code of Conduct. Also, check out some of our community resources including:
- Community Wiki
- Salt’s Contributor Guide
- Join our Community Slack
- IRC on LiberaChat
- Salt Project YouTube channel
- Salt Project Twitch channel
There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar. If you have additional questions, email us at [email protected]. We’re glad you’ve joined our community and look forward to doing awesome things with you!
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
I am also having this issue on salt 3006.7 with salt-pip install mysql. And upgrade to 3007.0 doesn't fix anything.
$ salt-minion -V
Salt Version:
Salt: 3007.0
Python Version:
Python: 3.10.13 (main, Feb 19 2024, 03:31:20) [GCC 11.2.0]
Dependency Versions:
cffi: 1.16.0
cherrypy: 18.8.0
dateutil: 2.8.2
docker-py: 5.0.3
gitdb: Not Installed
gitpython: Not Installed
Jinja2: 3.1.3
libgit2: Not Installed
looseversion: 1.3.0
M2Crypto: Not Installed
Mako: Not Installed
msgpack: 1.0.7
msgpack-pure: Not Installed
mysql-python: Not Installed
packaging: 23.1
pycparser: 2.21
pycrypto: Not Installed
pycryptodome: 3.19.1
pygit2: Not Installed
python-gnupg: 0.5.2
PyYAML: 6.0.1
PyZMQ: 25.1.2
relenv: 0.15.1
smmap: Not Installed
timelib: 0.3.0
Tornado: 6.3.3
ZMQ: 4.3.4
Salt Package Information:
Package Type: onedir
System Versions:
dist: ubuntu 22.04.3 jammy
locale: utf-8
machine: x86_64
release: 6.5.0-1016-gcp
system: Linux
version: Ubuntu 22.04.3 jammy
End of the logs from sudo salt-pip install mysql -v
Collecting mysqlclient (from mysql)
Using cached mysqlclient-2.2.4.tar.gz (90 kB)
Running command pip subprocess to install build dependencies
Collecting setuptools>=61
Using cached setuptools-69.2.0-py3-none-any.whl.metadata (6.3 kB)
Using cached setuptools-69.2.0-py3-none-any.whl (821 kB)
Installing collected packages: setuptools
Successfully installed setuptools-69.2.0
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
Installing build dependencies ... done
Running command Getting requirements to build wheel
Error in sitecustomize; set PYTHONVERBOSE for traceback:
AssertionError:
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
main()
File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
execsitecustomize()
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 959, in wrapper
import sitecustomize
File "/tmp/pip-build-env-nyva4fg7/site/sitecustomize.py", line 22, in <module>
assert not path in sys.path
AssertionError
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
full command: /opt/saltstack/salt/bin/python3.10 /opt/saltstack/salt/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpf_vj3h96
cwd: /tmp/pip-install-8_ym37sk/mysqlclient_a2565e4f337c4ec39f5c333f8a3c1fa1
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
Just adding. Installing mysql in another python virtualenv didn't have any effect on installing it in the salt python env. I have not been able to move past this issue.
I have the same issue installing mysqlclient in 3007.0 on Debian 11:
root@host:~ # salt-pip install mysqlclient
Collecting mysqlclient
Downloading mysqlclient-2.2.4.tar.gz (90 kB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 90.4/90.4 kB 4.2 MB/s eta 0:00:00
Installing build dependencies ... done
Getting requirements to build wheel ... error
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
Error in sitecustomize; set PYTHONVERBOSE for traceback:
AssertionError:
Fatal Python error: init_import_site: Failed to import the site module
Python runtime state: initialized
Traceback (most recent call last):
File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
main()
File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
execsitecustomize()
File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 959, in wrapper
import sitecustomize
File "/tmp/pip-build-env-mcidy7d6/site/sitecustomize.py", line 22, in <module>
assert not path in sys.path
AssertionError
[end of output]
note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error
× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.
note: This error originates from a subprocess, and is likely not a problem with pip.
[notice] A new release of pip is available: 23.3.2 -> 24.0
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
I tried upgrading pip, but that didn't help. I modified the file lib/python3.10/site-packages/pip/_internal/build_env.py and commented the line with "assert not path in sys.path" and retried, and that gave different errors. It complained about missing mysqlclient dev libraries and pkg-config not being installed, which I resolved (using apt-get install pkg-config libmariadb-dev-compat) and then salt-pip install mysqlclient worked. Not sure what the impact of removing the assert is, this is just a test minion so fine for now but would like a proper fix. But at least the salt states using the mysql_ modules work after this hack.
seems similar to https://github.com/saltstack/salt/issues/65980 and https://github.com/saltstack/salt/issues/65972 TLDR => workaround :
salt-pip install pip==22.3.1
salt-pip install <package>
source : salt-pip install
My "solution" mentioned in the other post is to apply a patch salt-pip itself:
https://github.com/saltstack/salt/issues/65980#issuecomment-2084731676
Patch and Instructions (as well as a LONG Explanation / Analysis / Test) is provided in my comment ABOVE the mentioned link.
For those who are interested, I was able to track this down to an issue in relenv: saltstack/relenv#182
I stumbled upon a flavor of this issue, combined with #35013 and #57339. I had upgraded salt-minions from 3004.x to 3007.x, and anywhere the x509 module was used (v1, not the new v2), I got the 'x509' __virtual__ returned False: Could not load x509 state: m2crypto unavailable error.
This was on Debian 11/12, already having python3-m2crypto installed. Trying to install it via salt-pip install m2crypto gave me the same error as experienced in this issue. There was also a few other dependencies that needed to be installed.
Final solution for me was;
salt-pip install pip==22.3.1
apt install -y libssl-dev swig python3-dev gcc
salt-pip install m2crypto
... or if you want to fix it via salt-master;
x509_fix_pip_module:
cmd.run:
- name: |
salt-pip install pip==22.3.1
apt install -y libssl-dev swig python3-dev gcc
salt-pip install m2crypto
- shell: /bin/bash
@dwoz Confirmed this is real with Ubuntu 22.04 on an AWS VM with Salt 3006.8, but checking libvirt on PyPI,
Requirements
To build the libvirt python binary requires the GCC and pkg-config tools, and development headers for the libvirt and python libraries.
So after installing the following:
apt install build-essential
apt install pkg-config
apt install libvirt-dev
The command salt-pip install libvirt and the result is successful
root@ip-172-31-2-109:/home/ubuntu# salt-pip list | grep libvirt
libvirt-python 10.5.0
[notice] A new release of pip is available: 23.3.2 -> 24.2
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip
root@ip-172-31-2-109:/home/ubuntu#
Seems it is working similar to regular pip, which salt-pip is working similarly,salt-pip and pip3 both need the requirements to install libvirt satisfied before the install can be successful.
Closing this since functioning as expected. Feel free to reopen if there is other information pertinent to the issue. Note: salt-pip duplicates pip3 functionality, pip installing the into the correct place for Salt onedir architecture, and has the same limitations as pip3, that is, if pip3 has requirements, then salt-pip has similar requirements to function correctly.
