Image builder fails to build RHEL 9 ova for vsphere

[gtomilko]

What happened: Run image builder: image-builder build --os redhat --os-version 9 --hypervisor vsphere --release-channel 1-31 --vsphere-config /home/somepath/vc.json --firmware bios --ansible-verbosity 3 --force

It goes through vm creation, OS install and reboot, After that it fails ansible run:

vsphere-iso.vsphere: ansible-playbook [core 2.15.13]
    vsphere-iso.vsphere:   config file = /root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/ansible.cfg
    vsphere-iso.vsphere:   configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
    vsphere-iso.vsphere:   ansible python module location = /root/.local/lib/python3.9/site-packages/ansible
    vsphere-iso.vsphere:   ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
    vsphere-iso.vsphere:   executable location = /root/.local/bin/ansible-playbook
    vsphere-iso.vsphere:   python version = 3.9.21 (main, Dec  5 2024, 00:00:00) [GCC 11.5.0 20240719 (Red Hat 11.5.0-2)] (/usr/bin/python3)
    vsphere-iso.vsphere:   jinja version = 3.1.6
    vsphere-iso.vsphere:   libyaml = True
    vsphere-iso.vsphere: Using /root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/ansible.cfg as config file
    vsphere-iso.vsphere: host_list declined parsing /tmp/packer-provisioner-ansible1826804724 as it did not pass its verify_file() method
    vsphere-iso.vsphere: script declined parsing /tmp/packer-provisioner-ansible1826804724 as it did not pass its verify_file() method
    vsphere-iso.vsphere: auto declined parsing /tmp/packer-provisioner-ansible1826804724 as it did not pass its verify_file() method
    vsphere-iso.vsphere: Parsed /tmp/packer-provisioner-ansible1826804724 inventory source with ini plugin
    vsphere-iso.vsphere: Skipping callback 'default', as we already have a stdout callback.
    vsphere-iso.vsphere: Skipping callback 'minimal', as we already have a stdout callback.
    vsphere-iso.vsphere: Skipping callback 'oneline', as we already have a stdout callback.
    vsphere-iso.vsphere:
    vsphere-iso.vsphere: PLAYBOOK: firstboot.yml ********************************************************
    vsphere-iso.vsphere: 2 plays in /root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/ansible/firstboot.yml
    vsphere-iso.vsphere:
    vsphere-iso.vsphere: PLAY [Bootstrap Python] ********************************************************
    vsphere-iso.vsphere: <127.0.0.1> ESTABLISH SSH CONNECTION FOR USER: builder
    vsphere-iso.vsphere: <127.0.0.1> SSH: EXEC ssh '-o IdentitiesOnly=yes' -o StrictHostKeyChecking=no -o Port=36081 -o 'IdentityFile="/tmp/ansible-key3475743021"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="builder"' -o ConnectTimeout=10 '-o IdentitiesOnly=yes' -tt 127.0.0.1 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-ntthhosudmoenfhwhjyrseqjczapubxg ; test -e "/usr/bin/python3"'"'"''
    vsphere-iso.vsphere: <127.0.0.1> (255, b'', b'Connection timed out during banner exchange\r\nConnection to 127.0.0.1 port 36081 timed out\r\n')
    vsphere-iso.vsphere:
    vsphere-iso.vsphere: TASK [Check if Python is installed] ********************************************
    vsphere-iso.vsphere: task path: /root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi/ansible/firstboot.yml:20
    vsphere-iso.vsphere: fatal: [default]: UNREACHABLE! => {
    vsphere-iso.vsphere:     "changed": false,
    vsphere-iso.vsphere:     "msg": "Failed to connect to the host via ssh: Connection timed out during banner exchange\r\nConnection to 127.0.0.1 port 36081 timed out",
    vsphere-iso.vsphere:     "unreachable": true
    vsphere-iso.vsphere: }
    vsphere-iso.vsphere:
    vsphere-iso.vsphere: PLAY RECAP *********************************************************************
    vsphere-iso.vsphere: default                    : ok=0    changed=0    unreachable=1    failed=0    skipped=0    rescued=0    ignored=0
    vsphere-iso.vsphere:
==> vsphere-iso.vsphere: Provisioner failed with "Error executing Ansible: Non-zero exit status: exit status 4", retrying with 4 trie(s) left
==> vsphere-iso.vsphere: Setting a 30m0s timeout for the next provisioner...
==> vsphere-iso.vsphere: Provisioner failed with "Error executing Ansible: Non-zero exit status: exit status 4", retrying with 3 trie(s) left
==> vsphere-iso.vsphere: Setting a 30m0s timeout for the next provisioner...
==> vsphere-iso.vsphere: Provisioner failed with "Error executing Ansible: Non-zero exit status: exit status 4", retrying with 2 trie(s) left
==> vsphere-iso.vsphere: Setting a 30m0s timeout for the next provisioner...
==> vsphere-iso.vsphere: Provisioner failed with "Error executing Ansible: Non-zero exit status: exit status 4", retrying with 1 trie(s) left
==> vsphere-iso.vsphere: Setting a 30m0s timeout for the next provisioner...
==> vsphere-iso.vsphere: retry limit reached.
==> vsphere-iso.vsphere: Provisioning step had errors: Running the cleanup provisioner, if present...
==> vsphere-iso.vsphere: Clear boot order...
==> vsphere-iso.vsphere: Power off VM...
==> vsphere-iso.vsphere: Destroying VM...
Build 'vsphere-iso.vsphere' errored after 6 minutes 33 seconds: Error executing Ansible: Non-zero exit status: exit status 4

==> Wait completed after 6 minutes 33 seconds

==> Some builds didn't complete successfully and had errors:
--> vsphere-iso.vsphere: Error executing Ansible: Non-zero exit status: exit status 4

==> Builds finished but no artifacts were created.
make[1]: *** [Makefile:476: build-node-ova-vsphere-rhel-9] Error 1
make[1]: Leaving directory '/root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder/image-builder/images/capi'
make: *** [Makefile:421: local-build-ova-redhat-9] Error 2
2025/03/17 19:03:48 Error executing image-builder for vsphere hypervisor: failed to run command: exit status 2
make: Leaving directory '/root/eks-anywhere-build-tooling/projects/kubernetes-sigs/image-builder'

What you expected to happen: Expected to get ova.

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• EKS Anywhere Release: 1-31
• EKS Distro Release: 0.22.5