kubespray Kubespray installation Fails. for "FAILED - RETRYING: [node2]: Download_container | Download image if required (1 retries left)"

Kubespray installation Fails. for "FAILED - RETRYING: [node2]: Download_container | Download image if required (1 retries left)"

Open Sayantan-Dell opened this issue 11 months ago • 3 comments

What happened?

Error Message while executing ansible-playbook command -

TASK [download : debug] ****************************************************************************************************** ok: [node1] => { "msg": "Pull registry.k8s.io/pause:3.9 required is: True" } ok: [node2] => { "msg": "Pull registry.k8s.io/pause:3.9 required is: True" } ok: [node3] => { "msg": "Pull registry.k8s.io/pause:3.9 required is: True" } Monday 25 March 2024 15:31:30 +0000 (0:00:00.052) 0:03:20.513 ********** Monday 25 March 2024 15:31:30 +0000 (0:00:00.046) 0:03:20.560 ********** Monday 25 March 2024 15:31:30 +0000 (0:00:00.039) 0:03:20.599 ********** Monday 25 March 2024 15:31:30 +0000 (0:00:00.039) 0:03:20.638 ********** FAILED - RETRYING: [node2]: Download_container | Download image if required (4 retries left). FAILED - RETRYING: [node3]: Download_container | Download image if required (4 retries left). FAILED - RETRYING: [node1]: Download_container | Download image if required (4 retries left). FAILED - RETRYING: [node3]: Download_container | Download image if required (3 retries left). FAILED - RETRYING: [node1]: Download_container | Download image if required (3 retries left). FAILED - RETRYING: [node2]: Download_container | Download image if required (3 retries left). FAILED - RETRYING: [node3]: Download_container | Download image if required (2 retries left). FAILED - RETRYING: [node1]: Download_container | Download image if required (2 retries left). FAILED - RETRYING: [node3]: Download_container | Download image if required (1 retries left). FAILED - RETRYING: [node2]: Download_container | Download image if required (2 retries left).

TASK [download : Download_container | Download image if required] ************************************************************ fatal: [node3]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "registry.k8s.io/pause:3.9"], "delta": "0:00:00.453319", "end": "2024-03-25 15:31:48.673028", "msg": "non-zero return code", "rc": 1, "start": "2024-03-25 15:31:48.219709", "stderr": "time="2024-03-25T15:31:48Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io\nctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority", "stderr_lines": ["time="2024-03-25T15:31:48Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io", "ctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority"], "stdout": "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) "]} FAILED - RETRYING: [node1]: Download_container | Download image if required (1 retries left). FAILED - RETRYING: [node2]: Download_container | Download image if required (1 retries left). fatal: [node1]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "registry.k8s.io/pause:3.9"], "delta": "0:00:00.456418", "end": "2024-03-25 15:31:57.548947", "msg": "non-zero return code", "rc": 1, "start": "2024-03-25 15:31:57.092529", "stderr": "time="2024-03-25T15:31:57Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io\nctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority", "stderr_lines": ["time="2024-03-25T15:31:57Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io", "ctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority"], "stdout": "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) "]} fatal: [node2]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "--hosts-dir", "/etc/containerd/certs.d", "registry.k8s.io/pause:3.9"], "delta": "0:00:00.451734", "end": "2024-03-25 15:32:01.436287", "msg": "non-zero return code", "rc": 1, "start": "2024-03-25 15:32:00.984553", "stderr": "time="2024-03-25T15:32:01Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io\nctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority", "stderr_lines": ["time="2024-03-25T15:32:01Z" level=info msg="trying next host" error="failed to do request: Head \"https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9\": tls: failed to verify certificate: x509: certificate signed by unknown authority" host=registry.k8s.io", "ctr: failed to resolve reference "registry.k8s.io/pause:3.9": failed to do request: Head "https://europe-west2-docker.pkg.dev/v2/k8s-artifacts-prod/images/pause/manifests/3.9": tls: failed to verify certificate: x509: certificate signed by unknown authority"], "stdout": "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.1 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.2 s total: 0.0 B (0.0 B/s) \nregistry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| \nelapsed: 0.3 s total: 0.0 B (0.0 B/s) ", "stdout_lines": ["registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.1 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.2 s total: 0.0 B (0.0 B/s) ", "registry.k8s.io/pause:3.9: resolving |\u001b[32m\u001b[0m--------------------------------------| ", "elapsed: 0.3 s total: 0.0 B (0.0 B/s) "]}

NO MORE HOSTS LEFT ***********************************************************************************************************

PLAY RECAP ******************************************************************************************************************* node1 : ok=365 changed=57 unreachable=0 failed=1 skipped=439 rescued=0 ignored=0 node2 : ok=337 changed=57 unreachable=0 failed=1 skipped=380 rescued=0 ignored=0 node3 : ok=302 changed=51 unreachable=0 failed=1 skipped=372 rescued=0 ignored=0

What did you expect to happen?

I saw in other issues this issue was resolved using nerdctl updated version. I have checked the .yml files and can also see downloaded files are properly getting copied to usr/local/bin folder but still this issue is occurring. Can anyone suggest a Fix

How can we reproduce it (as minimally and precisely as possible)?

We have followed the installation guide . Commands used described below. Have used this to create 3 Node Cluster.

42 git clone https://github.com/kubernetes-sigs/kubespray.git 43 ls -ltr 44 cd kubespray 45 ls -ltr 46 sudo pip install -r requirements.txt 47 pwd 48 ansible --version 49 cp -rfp inventory/sample inventory/mycluster 50 pwd 51 cd inventory 52 ls -ltr 53 cd mycluster 54 ls -ltr 55 cat inventory.ini 56 cd ../../ 57 pwd 58 declare -a IPS=(10.48.148.224 10.48.148.225 10.48.148.226) 59 CONFIG_FILE=inventory/mycluster/hosts.yml python3 contrib/inventory_builder/inventory.py ${IPS[@]} 60 vi inventory/mycluster/hosts.yaml 61 vi inventory/mycluster/hosts.yml 62 pwd 63 vi inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml 64 ls -ltr | hrep ^d 65 ls -ltr | grep ^d 66 vi inventory/mycluster/group_vars/k8s_cluster/addons.yml 67 grep -i "true" inventory/mycluster/group_vars/k8s_cluster/addons.yml 68 grep -i "true" inventory/mycluster/group_vars/k8s_cluster/addons.yml | grep -v "#" 69 pwd 70 grep -v "#" inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml 71 pwd 72 ansible all -i inventory/mycluster/hosts.yml -m shell -a "sudo systemctl stop firewalld && sudo systemctl disable firewalld" 73 ansible all -i inventory/mycluster/hosts.yaml -m shell -a "echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf" 74 ansible all -i inventory/mycluster/hosts.yml -m shell -a "echo 'net.ipv4.ip_forward=1' | sudo tee -a /etc/sysctl.conf" 75 ansible all -i inventory/mycluster/hosts.yml -m shell -a "sudo sed -i '/ swap / s/^(.*)$/#\1/g' /etc/fstab && sudo swapoff -a" 76 pwd 77 ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

OS

user@node1:~$ lsb_release -d Description: Ubuntu 22.04.4 LTS

Version of Ansible

user@node1:~$ ansible --version ansible [core 2.16.4] config file = None configured module search path = ['/home/user/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/local/lib/python3.10/dist-packages/ansible ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections executable location = /usr/local/bin/ansible python version = 3.10.12 (main, Nov 20 2023, 15:14:05) [GCC 11.4.0] (/usr/bin/python3) jinja version = 3.1.2 libyaml = True

Version of Python

python - 3.10.12 / PIP - 22.0.2

Version of Kubespray (commit)

user@mgmt1:~/kubespray$ git rev-parse --short HEAD 2cb8c8544

Network plugin used

calico

Full inventory with variables

user@mgmt1:~/kubespray$ user@mgmt1:~/kubespray$ ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]" [WARNING]: provided hosts list is empty, only localhost is available. Note that the implicit localhost does not match 'all' [WARNING]: Skipping callback plugin 'ara_default', unable to load

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yml --become --become-user=root cluster.yml

Output of ansible run

NO MORE HOSTS LEFT ***********************************************************************************************************

Anything else we need to know

No response

Mar 26 '24 06:03 Sayantan-Dell

kubespray kubespray copied to clipboard

Kubespray installation Fails. for "FAILED - RETRYING: [node2]: Download_container | Download image if required (1 retries left)"

What happened?

What did you expect to happen?

How can we reproduce it (as minimally and precisely as possible)?

OS

Version of Ansible

Version of Python

Version of Kubespray (commit)

Network plugin used

Full inventory with variables

Command used to invoke ansible

Output of ansible run

Anything else we need to know

kubespray
kubespray copied to clipboard