local_path_provisioner_enabled: true - error

[safetylab]

What happened?

TASK [download : Download_container | Download image if required] **************************************************************************************************************************************************** fatal: [worker-2]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "rancher/local-path-provisioner:v0.0.19"], "delta": "0:00:00.055235", "end": "2024-04-30 22:05:49.405116", "msg": "non-zero return code", "rc": 1, "start": "2024-04-30 22:05:49.349881", "stderr": "time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher\nctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host", "stderr_lines": ["time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher", "ctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host"], "stdout": "", "stdout_lines": []}

What did you expect to happen?

local_path_provisioner_enabled: true local_path_provisioner_namespace: "local-path-storage" local_path_provisioner_storage_class: "local-path" local_path_provisioner_reclaim_policy: Delete local_path_provisioner_claim_root: /opt/local-path-provisioner/ local_path_provisioner_debug: false local_path_provisioner_image_repo: "rancher/local-path-provisioner" local_path_provisioner_image_tag: "v0.0.19" local_path_provisioner_helper_image_repo: "busybox" local_path_provisioner_helper_image_tag: "latest"

How can we reproduce it (as minimally and precisely as possible)?

cat k8s-cluster.yml

kube_config_dir: /etc/kubernetes kube_script_dir: "{{ bin_dir }}/kubernetes-scripts" kube_manifest_dir: "{{ kube_config_dir }}/manifests" kube_cert_dir: "{{ kube_config_dir }}/ssl" kube_token_dir: "{{ kube_config_dir }}/tokens" kube_api_anonymous_auth: true kube_version: v1.29.3 local_release_dir: "/tmp/releases" retry_stagger: 5 kube_cert_group: kube-cert kube_log_level: 2 credentials_dir: "{{ inventory_dir }}/credentials" kube_network_plugin: calico kube_network_plugin_multus: false kube_service_addresses: 10.233.0.0/18 kube_pods_subnet: 10.233.64.0/18 kube_network_node_prefix: 24 calico_pool_blocksize: 24 enable_dual_stack_networks: false kube_apiserver_ip: "{{ kube_service_addresses|ipaddr('net')|ipaddr(1)|ipaddr('address') }}" kube_apiserver_port: 6443 # (https) kube_apiserver_insecure_port: 0 # (disabled) kube_proxy_mode: iptables kube_proxy_strict_arp: false kube_proxy_nodeport_addresses: >- {%- if kube_proxy_nodeport_addresses_cidr is defined -%} [{{ kube_proxy_nodeport_addresses_cidr }}] {%- else -%} [] {%- endif -%}

kube_encrypt_secret_data: false cluster_name: cluster.local ndots: 2 dns_mode: coredns enable_nodelocaldns: true enable_nodelocaldns_secondary: false nodelocaldns_ip: 10.233.0.4 nodelocaldns_health_port: 9254 nodelocaldns_second_health_port: 9256 nodelocaldns_bind_metrics_host_ip: false nodelocaldns_secondary_skew_seconds: 5 enable_coredns_k8s_external: false coredns_k8s_external_zone: k8s_external.local enable_coredns_k8s_endpoint_pod_names: false resolvconf_mode: host_resolvconf deploy_netchecker: false skydns_server: "{{ kube_service_addresses|ipaddr('net')|ipaddr(3)|ipaddr('address') }}" skydns_server_secondary: "{{ kube_service_addresses|ipaddr('net')|ipaddr(4)|ipaddr('address') }}" dns_domain: "{{ cluster_name }}"

kata_containers_enabled: false kubeadm_certificate_key: "{{ lookup('password', credentials_dir + '/kubeadm_certificate_key.creds length=64 chars=hexdigits') | lower }}" k8s_image_pull_policy: IfNotPresent kubernetes_audit: false dynamic_kubelet_configuration: false default_kubelet_config_dir: "{{ kube_config_dir }}/dynamic_kubelet_dir" dynamic_kubelet_configuration_dir: "{{ kubelet_config_dir | default(default_kubelet_config_dir) }}" podsecuritypolicy_enabled: false

Amount of time to retain events. (default 1h0m0s)

event_ttl_duration: "1h0m0s"

Automatically renew K8S control plane certificates on first Monday of each month

auto_renew_certificates: false

First Monday of each month

auto_renew_certificates_systemd_calendar: "Mon --1,2,3,4,5,6,7 03:{{ groups['kube_control_plane'].index(inventory_hostname) }}0:00"

local_path_provisioner_enabled: true
local_path_provisioner_namespace: "local-path-storage"
local_path_provisioner_storage_class: "local-path"
local_path_provisioner_reclaim_policy: Delete
local_path_provisioner_claim_root: /opt/local-path-provisioner/
local_path_provisioner_debug: false
local_path_provisioner_image_repo: "rancher/local-path-provisioner"
local_path_provisioner_image_tag: "v0.0.24"
local_path_provisioner_helper_image_repo: "busybox"
local_path_provisioner_helper_image_tag: "latest"

local_volume_provisioner_enabled: true
local_volume_provisioner_namespace: kube-system
local_volume_provisioner_nodelabels:

• kubernetes.io/hostname
  local_volume_provisioner_storage_classes:
  local-storage:
  host_dir: /mnt/data
  mount_dir: /mnt/data
  volume_mode: Filesystem
  fs_type: ext4

OS

ubuntu22

Version of Ansible

2.16.6

Version of Python

3.12

Version of Kubespray (commit)

main and v2.24.1

Network plugin used

calico

Full inventory with variables

ansible-playbook upgrade-cluster.yml -b -i inventory/files/inventory.ini -e kube_version=v1.29.3 --limit "master-1:worker-2:worker-3"

Command used to invoke ansible

none

Output of ansible run

TASK [download : Download_container | Download image if required] **************************************************************************************************************************************************** fatal: [worker-2]: FAILED! => {"attempts": 4, "changed": true, "cmd": ["/usr/local/bin/ctr", "-n", "k8s.io", "images", "pull", "rancher/local-path-provisioner:v0.0.19"], "delta": "0:00:00.055235", "end": "2024-04-30 22:05:49.405116", "msg": "non-zero return code", "rc": 1, "start": "2024-04-30 22:05:49.349881", "stderr": "time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher\nctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host", "stderr_lines": ["time="2024-04-30T22:05:49Z" level=info msg="trying next host" error="failed to do request: Head \"https://rancher/v2/local-path-provisioner/manifests/v0.0.19\": dial tcp: lookup rancher on 8.8.8.8:53: no such host" host=rancher", "ctr: failed to resolve reference "rancher/local-path-provisioner:v0.0.19": failed to do request: Head "https://rancher/v2/local-path-provisioner/manifests/v0.0.19": dial tcp: lookup rancher on 8.8.8.8:53: no such host"], "stdout": "", "stdout_lines": []}

Anything else we need to know

No response

[wandersonlima]

I think the variable local_path_provisioner_image_repo: "rancher/local-path-provisioner" should be local_path_provisioner_image_repo: "{{ docker_image_repo }}/rancher/local-path-provisioner"

[safetylab]

Really? kubespray/inventory/sample/group_vars/k8s_cluster/addons.yml - local_path_provisioner_image_repo: "rancher/local-path-provisioner" - is it wrong in sample?

[akoken]

Really? kubespray/inventory/sample/group_vars/k8s_cluster/addons.yml - local_path_provisioner_image_repo: "rancher/local-path-provisioner" - is it wrong in sample?

Update as "docker.io/rancher/local-path-provisioner"

[safetylab]

Really? kubespray/inventory/sample/group_vars/k8s_cluster/addons.yml - local_path_provisioner_image_repo: "rancher/local-path-provisioner" - is it wrong in sample?

Update as "docker.io/rancher/local-path-provisioner"

yes works