salt icon indicating copy to clipboard operation
salt copied to clipboard

Published 20 hours ago • verified publisher icon saltstack

virt module (libvirt) broken on EL9

Open gandihar opened this issue 1 year ago • 10 comments

Description Somehow, the virt (libvirt) module is broken on EL9. It works on EL7-8 but fails with: 'virt' __virtual__ returned False: Unable to locate or import python libvirt library.

Setup Using the onedir Salt Minion packages from the official repo (3006.). Used 'pip.install' as per Documents to install the libvirt-python module. Dependencies (libvirt client, libvirt libs, grep) are pressent. Issuing any 'virt.' commands returns the error 'virt' __virtual__ returned False: Unable to locate or import python libvirt library. on any EL9 variant, but works on EL7 and 8.

  • [X] on-prem machine
  • [X] onedir packaging
  • [X] used bootstrap to install

Steps to Reproduce the behavior

  • Install Salt-minion using the onedir RPMs in the official repo on an EL9 system (Tested OEL9, Alma9, Rocky9).
  • Use Salt States to pip.install libvirt-python.
  • Issue 'virt.' commands, or use 'virt.' States. 'virt' __virtual__ returned False: Unable to locate or import python libvirt library. ( There are no logs, even using -l debug )

Expected behavior The output of the 'virt.*' command or State that is issued.

Versions Report Master

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: centos 7.9.2009 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.102.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7.9.2009 Core

Working Minion (EL7)

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: centos 7.9.2009 Core
        locale: utf-8
       machine: x86_64
       release: 3.10.0-1160.102.1.el7.x86_64
        system: Linux
       version: CentOS Linux 7.9.2009 Core

local:
    ----------
    libvirt-python:
        9.4.0

Working Minion (EL8)

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: almalinux 8.9 Midnight Oncilla
        locale: utf-8
       machine: x86_64
       release: 4.18.0-513.9.1.el8_9.x86_64
        system: Linux
       version: AlmaLinux 8.9 Midnight Oncilla

local:
    ----------
    libvirt-python:
        9.4.0

Broken Minion (EL9)

Salt Version:
          Salt: 3006.5
 
Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]
 
Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4
 
System Versions:
          dist: almalinux 9.3 Shamrock Pampas Cat
        locale: utf-8
       machine: x86_64
       release: 5.14.0-362.8.1.el9_3.x86_64
        system: Linux
       version: AlmaLinux 9.3 Shamrock Pampas Cat

local:
    ----------
    libvirt-python:
        9.10.0

(libvirt-python removed and re-installed as part of troubleshooting) (the new version, it does nothing)

gandihar avatar Dec 13 '23 20:12 gandihar

Did you use salt-pip to install libvirt?

/opt/saltstack/salt/salt-pip install libvirt-python

dwoz avatar Dec 13 '23 23:12 dwoz

'Sup Woz!

Did you use salt-pip to install libvirt?

/opt/saltstack/salt/salt-pip install libvirt-python

The pip install is part of a State that I apply to all my vm hosts:

libvirt-packages:
  pkg.installed:
    - pkgs: {{ vserver.pkgs | yaml }}
    - require:
      - mount: mount-vms
      - mount: mount-imgs

libvirt-pip:
  pip.installed:
    - name: libvirt-python
    - reload_modules: true
    - require:
      - pkg: libvirt-packages

virt.pkgs comes from a map file, and varies slightly by osmajorrelease, but includes all of the following: - libvirt - libvirt-client - libvirt-daemon - qemu-kvm - virt-install - libvirt-devel

Oh, and also:

(el9) $ salt-pip install libvirt-python
Requirement already satisfied: libvirt-python in /opt/saltstack/salt/extras-3.10 (9.10.0)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv

[notice] A new release of pip available: 22.3.1 -> 23.3.1
[notice] To update, run: /opt/saltstack/salt/bin/python3.10 -m pip install --upgrade pip

gandihar avatar Dec 13 '23 23:12 gandihar

Can you successfully import eh necessary bits?

/opt/saltstack/salt/bin/python3 -c 'import libvirt; from libvirt import libvirtError'

dwoz avatar Dec 16 '23 11:12 dwoz

Okay, now were're getting somewhere:

$ /opt/saltstack/salt/bin/python3 -c 'import libvirt; from libvirt import libvirtError'
Traceback (most recent call last):
  File "/opt/saltstack/salt/extras-3.10/libvirt.py", line 16, in <module>
    import cygvirtmod as libvirtmod  # type: ignore
ModuleNotFoundError: No module named 'cygvirtmod'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/extras-3.10/libvirt.py", line 19, in <module>
    raise lib_e
  File "/opt/saltstack/salt/extras-3.10/libvirt.py", line 13, in <module>
    import libvirtmod  # type: ignore
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

gandihar avatar Dec 17 '23 00:12 gandihar

Confirmed the above is the same on Rocky 9 as well.

gandihar avatar Dec 17 '23 01:12 gandihar

Can you please provide the verbose output from installing libvirt?

/opt/saltsack/salt/salt-pip install -v libvirt-python

dwoz avatar Dec 17 '23 01:12 dwoz

/opt/saltstack/salt/salt-pip install -v libvirt-python (Took me a few tries to catch the typo) attached. salt-pip-install-libvirt.log

gandihar avatar Dec 18 '23 00:12 gandihar

lol, heads up, this is broken even harder in 3006.6:

EDIT: Actually, I am pretty sure any python module using Wheels is probably going to break (#65980) in 3006.6.

          ID: Ensure Python Modules for libvirt (Common)
    Function: pip.installed
        Name: libvirt-python
      Result: False
     Comment: Failed to install packages: libvirt-python. Error: Collecting libvirt-python
                Using cached libvirt-python-10.0.0.tar.gz (246 kB)
                Installing build dependencies: started
                Installing build dependencies: finished with status 'done'
                Getting requirements to build wheel: started
                Getting requirements to build wheel: finished with status 'error'   error: subprocess-exited-with-error

                × Getting requirements to build wheel did not run successfully.
                │ exit code: 1
                ╰─> [20 lines of output]
                    Error in sitecustomize; set PYTHONVERBOSE for traceback:
                    TypeError: expected str, bytes or os.PathLike object, not NoneType
                    Fatal Python error: init_import_site: Failed to import the site module
                    Python runtime state: initialized
                    Traceback (most recent call last):
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 617, in <module>
                        main()
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 610, in main
                        execsitecustomize()
                      File "/opt/saltstack/salt/lib/python3.10/site-packages/relenv/runtime.py", line 932, in wrapper
                        import sitecustomize
                      File "/tmp/pip-build-env-_qqckrye/site/sitecustomize.py", line 23, in <module>
                        site.addsitedir(path)
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 216, in addsitedir
                        sitedir, sitedircase = makepath(sitedir)
                      File "/opt/saltstack/salt/lib/python3.10/site.py", line 97, in makepath
                        dir = os.path.join(*paths)
                      File "/opt/saltstack/salt/lib/python3.10/posixpath.py", line 76, in join
                        a = os.fspath(a)
                    TypeError: expected str, bytes or os.PathLike object, not NoneType
                    [end of output]

                note: This error originates from a subprocess, and is likely not a problem with pip.
              error: subprocess-exited-with-error

              × Getting requirements to build wheel did not run successfully.
              │ exit code: 1
              ╰─> See above for output.

              note: This error originates from a subprocess, and is likely not a problem with pip.
     Started: 16:49:43.736838
    Duration: 4498.665 ms
     Changes:

nf-brentsaner avatar Feb 05 '24 17:02 nf-brentsaner

Have not directly confirmed for this module, but likely that this issue will be resolved with https://github.com/saltstack/relenv/issues/182 resolution.

smarsching did some great work on this in #65980 on tracking down where exactly the issue seems to occur and has some proposed long-term fixes.

nf-brentsaner avatar May 28 '24 20:05 nf-brentsaner

Yes this should be resolved in the next 3006.x release.

dwoz avatar Jun 19 '24 05:06 dwoz

This goes into 3006.9.

dwoz avatar Jul 29 '24 21:07 dwoz

Has anyone confirmed that the virt module and state is working in 3006.9? I am seeing it with the same issue in 3006.9, but it is functioning in 3006.1.

3006.9

[root@jpphype1 ~]# salt-call --versions
Salt Version:
          Salt: 3006.9

Python Version:
        Python: 3.10.14 (main, Jun 26 2024, 11:44:37) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
  cryptography: 42.0.5
      dateutil: 2.8.1
     docker-py: 7.1.0
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.4
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.19.1
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.17.0
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: oracle 9.4
        locale: utf-8
       machine: x86_64
       release: 5.15.0-101.103.2.1.el9uek.x86_64
        system: Linux
       version: Oracle Linux Server 9.4

[root@jpphype1 ~]# salt-call virt.is_hyper
/opt/saltstack/salt/lib/python3.10/site-packages/salt/utils/psutil_compat.py:16: DeprecationWarning: Please stop importing 'salt.utils.psutil_compat' and instead import 'psutil' directly as there's no longer a need for a compatability layer. The 'salt.utils.psutil_compat' will go away on Salt 3008.0 (Argon).
  salt.utils.versions.warn_until(
'virt' __virtual__ returned False: Unable to locate or import python libvirt library.

[root@jpphype1 ~]# /opt/saltstack/salt/bin/python3 -c 'import libvirt; from libvirt import libvirtError'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 16, in <module>
    import cygvirtmod as libvirtmod  # type: ignore
ModuleNotFoundError: No module named 'cygvirtmod'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 19, in <module>
    raise lib_e
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 13, in <module>
    import libvirtmod  # type: ignore
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

[root@jpphype1 ~]# /opt/saltstack/salt/bin/python3.10 -c 'import libvirt; from libvirt import libvirtError'
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 16, in <module>
    import cygvirtmod as libvirtmod  # type: ignore
ModuleNotFoundError: No module named 'cygvirtmod'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 19, in <module>
    raise lib_e
  File "/opt/saltstack/salt/lib/python3.10/site-packages/libvirt.py", line 13, in <module>
    import libvirtmod  # type: ignore
ImportError: /lib64/libldap.so.2: undefined symbol: EVP_md2, version OPENSSL_3.0.0

3006.1

[root@jpphype1 ~]# salt-call --versions
Salt Version:
          Salt: 3006.1

Python Version:
        Python: 3.10.11 (main, May  5 2023, 02:31:54) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: 7.1.0
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 5.4.1
         PyZMQ: 23.2.0
        relenv: 0.12.3
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: oracle 9.4
        locale: utf-8
       machine: x86_64
       release: 5.15.0-101.103.2.1.el9uek.x86_64
        system: Linux
       version: Oracle Linux Server 9.4

[root@jpphype1 ~]# salt-call virt.is_hyper
[INFO    ] Executing command 'ps' in directory '/root'
local:
    True

[root@jpphype1 ~]# /opt/saltstack/salt/bin/python3 -c 'import libvirt; from libvirt import libvirtError'

[root@jpphype1 ~]# /opt/saltstack/salt/bin/python3.10 -c 'import libvirt; from libvirt import libvirtError'

m0duspwnens avatar Aug 07 '24 17:08 m0duspwnens