VMs on hypervisors don't start

[praiskup]

    TASK [Activate Red Hat Subscription] ************
    Saturday 10 December 2022  19:52:08 +0000 (0:00:00.037)       0:00:04.216 *****
    DEBUG:root: -> exit_status=124, time=600.008s
    ....

This happens on all x86 and Power8 hypervisors (6 machines) but not on Power9.

[FrostyX]

My guess is that this is the same issue as #2426 (empty /etc/resolv.conf)

[FrostyX]

I also suspect this to be the same issue https://bugzilla.redhat.com/show_bug.cgi?id=2152305

[FrostyX]

The minimal reproducer here is

[root@copr-hv-ppc64le-02-dev-00001551-20221213-190955 ~]# curl subscription.rhsm.redhat.com:443
curl: (7) Couldn't connect to server

The TASK [Activate Red Hat Subscription] hangs infinitely because of this error because we are running it in loop - infinitely.

Edit: A known issue https://bugzilla.redhat.com/show_bug.cgi?id=2100803 However there reportedly never was an IPv6 address on that server, so something had to change for us during the F37 upgrade.

[FrostyX]

There was a misunderstanding on my side. I was talking about IPv6-only builders but we don't have any IPv6-only builders. We only have builders that don't have a public IPv4 address, only an IPv6 address. They still have a cloud-init eth0 connection though, which is IPv4 from libvirt.

This whole issue was caused by builders not properly activating this eth0 connection. With @praiskup, we think that some package was upgraded on our hypervisors and it made the libvirt network behave strangely.

And the solution after three weeks was just anticlimactic. We did systemctl restart libvirtd on all hypervisors and everything works as expected now.