oVirt
hosted-engine --deploy fails, "this is probably a bug"
oVirt 4.5.4 OS: Centos stream 8 and Centos stream 9
I am trying to deploy the hosted engine with
hosted-engine --deploy --4
After the step where I set the gateway address, the script crashes with:
[ INFO ] Checking available network interfaces: [ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n' [ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n' [ ERROR ] b"import name 'Callable' from 'collections'\n" [ ERROR ] b'(/usr/lib64/python3.11/collections/init.py)\n' [ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/init.py)\n" [ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook [ INFO ] Stage: Clean up [ INFO ] Cleaning temporary resources [ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n' [ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n' [ ERROR ] b"import name 'Callable' from 'collections'\n" [ ERROR ] b'(/usr/lib64/python3.11/collections/init.py)\n' [ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/init.py)\n" [ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook [ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20230522102503.conf' [ INFO ] Stage: Pre-termination [ INFO ] Stage: Termination [ ERROR ] Hosted Engine deployment failed Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230522102455-imqfjz.log
centos stream 8 or 9 makes not difference.
Steps I did after OS install:
- dnf module enable virt:rhel (not needed on stream 9)
- dnf update
- dnf install centos-release-ovirt45
- dnf install ovirt-hosted-engine-setup
- hosted-engine --deploy --4
Is there anything I am missing? Please let me know if you need the whole log file.
Update:
I solved this one with changing
import Callable from collections
to
from collections.abc import Callable
in /usr/share/ovirt-hosted-engine-setup/he_ansible/callback_plugins/2_ovirt_logger.py but I am running into the next problem:
The ipaddr filter requires python's netaddr be installed on the ansible controller.
It seems that there is a problem with the python dependencies. When installing the ovirt-hosted-engine-setup, python3.11 is installed too. It seems that there are some modules missing. I solved this by installing python3.11-pip and running pip3.11 install netaddr but I ran into the same error when deploying the hosted engine because the module was missing there too.
Update: Managed to install the host and the engine. Had to manually install jmespath and netaddr with pip3.11 on the host. When deploying the engine vm itself, I had to do the same there before the engine-setup starts.
I can confirm this bug still exists
hosted-engine --deploy --4 --6
[ INFO ] Stage: Initializing
[ INFO ] Stage: Environment setup
During customization use CTRL-D to abort.
Continuing will configure this host for serving as hypervisor and will create a local VM with a running engine.
The locally running engine will be used to configure a new storage domain and create a VM there.
At the end the disk of the local VM will be moved to the shared storage.
Are you sure you want to continue? (Yes, No)[Yes]:
Configuration files:
Log file: /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230824150418-lnzws4.log
Version: otopi-1.10.3 (otopi-1.10.3-1.el9)
[ INFO ] Stage: Environment packages setup
[ INFO ] Stage: Programs detection
[ INFO ] Stage: Environment setup (late)
[ INFO ] Stage: Environment customization
--== STORAGE CONFIGURATION ==--
--== HOST NETWORK CONFIGURATION ==--
Please indicate the gateway IP address [192.168.1.254]:
[ INFO ] Checking available network interfaces:
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n"
[ ERROR ] Failed to execute stage 'Environment customization': Failed executing ansible-playbook
[ INFO ] Stage: Clean up
[ INFO ] Cleaning temporary resources
[ ERROR ] b'[WARNING]: Skipping plugin (/usr/share/ovirt-hosted-engine-\n'
[ ERROR ] b'setup/he_ansible/callback_plugins/2_ovirt_logger.py), cannot load: cannot\n'
[ ERROR ] b"import name 'Callable' from 'collections'\n"
[ ERROR ] b'(/usr/lib64/python3.11/collections/__init__.py)\n'
[ ERROR ] b"ERROR! Unexpected Exception, this is probably a bug: cannot import name 'Callable' from 'collections' (/usr/lib64/python3.11/collections/__init__.py)\n"
[ ERROR ] Failed to execute stage 'Clean up': Failed executing ansible-playbook
[ INFO ] Generating answer file '/var/lib/ovirt-hosted-engine-setup/answers/answers-20230824150423.conf'
[ INFO ] Stage: Pre-termination
[ INFO ] Stage: Termination
[ ERROR ] Hosted Engine deployment failed
Log file is located at /var/log/ovirt-hosted-engine-setup/ovirt-hosted-engine-setup-20230824150418-lnzws4.log
/etc/os-release
NAME="Rocky Linux"
VERSION="9.2 (Blue Onyx)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="9.2"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Rocky Linux 9.2 (Blue Onyx)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:9::baseos"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2032-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-9"
ROCKY_SUPPORT_PRODUCT_VERSION="9.2"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.2"
dnf list --installed ovirt*
dnf list --installed ovirt*
Installed Packages
ovirt-ansible-collection.noarch 3.1.2-1.el9 @centos-ovirt45
ovirt-host.x86_64 4.5.0-3.el9 @centos-ovirt45
ovirt-host-dependencies.x86_64 4.5.0-3.el9 @centos-ovirt45
ovirt-hosted-engine-ha.noarch 2.5.0-1.el9 @centos-ovirt45
ovirt-hosted-engine-setup.noarch 2.7.0-1.el9 @centos-ovirt45
ovirt-imageio-client.x86_64 2.5.0-1.el9 @centos-ovirt45
ovirt-imageio-common.x86_64 2.5.0-1.el9 @centos-ovirt45
ovirt-imageio-daemon.x86_64 2.5.0-1.el9 @centos-ovirt45
ovirt-openvswitch.noarch 2.17-1.el9 @centos-ovirt45
ovirt-openvswitch-ipsec.noarch 2.17-1.el9 @centos-ovirt45
ovirt-openvswitch-ovn.noarch 2.17-1.el9 @centos-ovirt45
ovirt-openvswitch-ovn-common.noarch 2.17-1.el9 @centos-ovirt45
ovirt-openvswitch-ovn-host.noarch 2.17-1.el9 @centos-ovirt45
ovirt-provider-ovn-driver.noarch 1.2.36-1.el9 @centos-ovirt45
ovirt-python-openvswitch.noarch 2.17-1.el9 @centos-ovirt45
ovirt-vmconsole.noarch 1.0.9-1.el9 @centos-ovirt45
ovirt-vmconsole-host.noarch 1.0.9-1.el9 @centos-ovirt45
Holy cow. The 'collections.abc' bug is still there. Is there anyway this can be patched soon ? Can I help do so ?
Holy cow. The 'collections.abc' bug is still there. Is there anyway this can be patched soon ? Can I help do so ?
~~This software does not seem maintained, so~~ I think you should try something else.
Holy cow. The 'collections.abc' bug is still there. Is there anyway this can be patched soon ? Can I help do so ?
This software does not seem maintained, so I think you should try something else.
Don't spread false news. This is being maintained, but we can't handle everything at a time ofcourse :)
About this issue, it's already fixed for more than a year? https://github.com/oVirt/ovirt-hosted-engine-setup/commit/0cf9fac2a0e19a822eff4df8bf771fc919148e01
And contained in ovirt-hosted-engine-setup-2.7.1-1.el9.
Use the latest version please :)
@dupondje I had no success with engine-deployment using either of below methods
This is my first time I am deploying oVirt. I chose it based on online documentation stating native FC-SAN support is available.
Please let me know if my deployment process is flawed or if this is a bug. Thanks!
- Stock ISO https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/ovirt-node-ng-installer-latest-el9.iso
- stock ISO with nightly repository
dnf copr enable -y ovirt/ovirt-master-snapshot centos-stream-9
dnf install -y ovirt-release-master
Attempted with single interface, earlier tried bond0 made of 4 interface LACP, and then br0, and then vlan ( mgmt )
# nmcli c s
NAME UUID TYPE DEVICE
eno1 e4d4aa8e-e000-4094-894e-b3a1d7efb35c ethernet eno1
lo 908c740e-57ed-420a-bc6f-d2b4af5719f7 loopback lo
virbr0 e13d5033-c4a5-4cd5-966d-8a5af263c3f5 bridge virbr0
vnet0 c5998b70-09f8-4b64-88d3-886853c5f490 tun vnet0
eno2 77949a15-d669-440a-bf7e-c9ad4a048d68 ethernet --
eno3 2c68e756-a5a8-4ec2-bded-d4a496afad90 ethernet --
eno4 716a4621-a979-485b-9c11-e4ada3f9b4af ethernet --
# ovs-vsctl show
7a725c09-2058-464e-a7da-66601a4647da
Bridge br-int
fail_mode: secure
datapath_type: system
Port br-int
Interface br-int
type: internal
ovs_version: "2.17.9"
# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 0c:c4:7a:c6:12:1e brd ff:ff:ff:ff:ff:ff
altname enp7s0
inet xxx.xx.xx.xxx/24 brd xxx.xx.xx.xxx scope global noprefixroute eno1
valid_lft forever preferred_lft forever
3: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 0c:c4:7a:c6:12:1f brd ff:ff:ff:ff:ff:ff
altname enp8s0
4: eno3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 0c:c4:7a:c6:16:d2 brd ff:ff:ff:ff:ff:ff
altname enp2s0f0
5: eno4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether 0c:c4:7a:c6:16:d3 brd ff:ff:ff:ff:ff:ff
altname enp2s0f1
6: ovs-system: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000
link/ether 56:3a:5d:d7:46:c5 brd ff:ff:ff:ff:ff:ff
7: br-int: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN group default qlen 1000
link/ether ce:45:35:10:a9:c8 brd ff:ff:ff:ff:ff:ff
8: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN group default qlen 1000
link/ipip 0.0.0.0 brd 0.0.0.0
28: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
link/ether 52:54:00:01:30:90 brd ff:ff:ff:ff:ff:ff
inet 192.168.222.1/24 brd 192.168.222.255 scope global virbr0
valid_lft forever preferred_lft forever
29: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master virbr0 state UNKNOWN group default qlen 1000
link/ether fe:16:3e:23:76:7d brd ff:ff:ff:ff:ff:ff
inet6 fe80::fc16:3eff:fe23:767d/64 scope link
valid_lft forever preferred_lft forever
Error1 -- looks like NAT or forwarding not happening
[ INFO ] TASK [ovirt.ovirt.engine_setup : Install oVirt Engine package]
[ ERROR ] fatal: [localhost -> 192.168.222.22]: FAILED! => {"changed": false, "msg": "Failed to download metadata for repo 'centos-ceph-pacific': Cannot prepare internal mirrorlist: Curl error (6): Couldn't resolve host name for http://mirrorlist.centos.org/?release=8-stream&arch=x86_64&repo=storage-ceph-pacific [Could not resolve host: mirrorlist.centos.org]", "rc": 1, "results": []}
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Sync on engine machine]
[ INFO ] changed: [localhost -> 192.168.222.22]
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Set destination directory path]
Error2
[ INFO ] TASK [ovirt.ovirt.hosted_engine_setup : Notify the user about a failure]
[ ERROR ] fatal: [localhost]: FAILED! => {"changed": false, "msg": "There was a failure deploying the engine on the local engine VM. The system may not be provisioned according to the playbook results: please check the logs for the issue, fix accordingly or re-deploy from scratch.\n"}
[ ERROR ] b'[DEPRECATION WARNING]: Encryption using the Python crypt module is deprecated. \n'
[ ERROR ] b'The Python crypt module is deprecated and will be removed from Python 3.13. \n'
[ ERROR ] b'Install the passlib library for continued encryption functionality. This \n'
[ ERROR ] b'feature will be removed in version 2.17. Deprecation warnings can be disabled \n'
[ ERROR ] b'by setting deprecation_warnings=False in ansible.cfg.\n'
[ ERROR ] Failed to execute stage 'Closing up': Failed executing ansible-playbook
Steps I followed
- Installed stock ISO https://resources.ovirt.org/pub/ovirt-4.5/iso/ovirt-node-ng-installer/ovirt-node-ng-installer-latest-el9.iso
- Configured Network ( I had no success with either LACP bond0 + br0 + mgmt VLAN, or just plain native IP on single interface )
- Storage is local for hosted-engine ( Later planned to migrate on FC-SAN )
- Provided Internet connectivity to node, Also all fqdn are resolvable through local dns
- dnf update -y
- dnf copr enable -y ovirt/ovirt-master-snapshot centos-stream-9 && dnf install -y ovirt-release-master
- hosted-engine --deploy --4
Few Observation
- cockpit not directly accessible using username: root and password
- Had to comment root account in
/etc/cockpit/disallowed-users - Unable to see "Virtualization Navigation link" in cockpit, looks like GUI way of host-engine deployment removed.
Attached Log
ovirt-hosted-engine-setup-20250530144810-cmse76.log
Holy cow. The 'collections.abc' bug is still there. Is there anyway this can be patched soon ? Can I help do so ?
~This software does not seem maintained, so~ I think you should try something else.
LOL. I've most certainly done so since 2023. While OVirt was an excellent solution at the time I've had to migrate to another solution since.
Perhaps one day OVirt will be an option yet again, it has lots to offer.