helm used http proxy to access kube-apiserver when installing kubelet-csr-approver (with temporary solution)

[andrewfung729]

What happened?

Hi Kubespray Team. I am new to k8s. Glad to use kubespray!

I found that when installing kubelet-csr-approver, due to the lack of 'no_proxy' env, helm uses the http proxy to access kube-apiserver. Our corporate proxy logged the requests like CONNECT lb-apiserver.kubernetes.local:6443.

For the security and privacy, I have hidden the username, IPs, proxy URL.

Source shortcut: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml

playbook log:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

What did you expect to happen?

It should be passed normally

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/test/kubespray/roles/helm-apps/tasks/main.yml:6
ok: [cp-1] => {
    "changed": false,
    "command": "/usr/local/bin/helm",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "status": null,
    "stderr": "",
    "stderr_lines": [],
    "stdout": "",
    "stdout_lines": []
}

How can we reproduce it (as minimally and precisely as possible)?

1. Using kube-vip for HA (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/kube-vip.md)
2. Enable kubelet_csr_approver_enabled (hardening: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/hardening.md)

kubelet_rotate_server_certificates: true

3. Enable proxy (using proxy: https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/docs/proxy.md)

http_proxy: "http://foobar"
https_proxy: "http://foobar"
# https_proxy_cert_file: ""
# no_proxy: ""
# download_validate_certs: False
# additional_no_proxy: ""

OS

PRETTY_NAME="Ubuntu 22.04.4 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.4 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy

Version of Ansible

ansible [core 2.15.9] config file = /home/user/test/kubespray/ansible.cfg configured module search path = ['/home/user/test/kubespray/library'] ansible python module location = /home/user/miniconda3/envs/kubespray/lib/python3.11/site-packages/ansible ansible collection location = /home/user/.ansible/collections:/usr/share/ansible/collections executable location = /home/user/miniconda3/envs/kubespray/bin/ansible python version = 3.11.8 (main, Feb 26 2024, 21:39:34) [GCC 11.2.0] (/home/user/miniconda3/envs/kubespray/bin/python) jinja version = 3.1.2 libyaml = True

Version of Python

Python 3.11.8

Version of Kubespray (commit)

2cb8c8544fc5aeff7e1bc0865e65837dd6e8eaca (v2.24.1)

Network plugin used

calico

Full inventory with variables

provided in the reproduce method before

Command used to invoke ansible

ansible-playbook -i inventory/mycluster/hosts.yaml --become --become-user=root --ask-become-pass -vvvvvv cluster.yml

Output of ansible run

playbook output:

TASK [helm-apps : Update Helm repositories] ************************************************************************************************************************
task path: /home/user/kubespray/roles/helm-apps/tasks/main.yml:30
fatal: [cp-1]: FAILED! => {
    "changed": false,
    "command": "/usr/local/bin/helm list --output=yaml --filter dummy",
    "invocation": {
        "module_args": {
            "api_key": null,
            "atomic": false,
            "binary_path": "/usr/local/bin/helm",
            "ca_cert": null,
            "chart_ref": null,
            "chart_repo_url": null,
            "chart_version": null,
            "context": null,
            "create_namespace": false,
            "dependency_update": false,
            "disable_hook": false,
            "force": false,
            "history_max": null,
            "host": null,
            "kubeconfig": null,
            "post_renderer": null,
            "purge": true,
            "release_name": "dummy",
            "release_namespace": "kube-system",
            "release_state": "absent",
            "release_values": {},
            "replace": false,
            "set_values": null,
            "skip_crds": false,
            "state": "absent",
            "timeout": null,
            "update_repo_cache": true,
            "validate_certs": true,
            "values_files": [],
            "wait": false,
            "wait_timeout": null
        }
    },
    "msg": "Failure when executing Helm command. Exited 1.\nstdout: \nstderr: Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr": "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority\n",
    "stderr_lines": [
        "Error: Kubernetes cluster unreachable: Get \"https://lb-apiserver.kubernetes.local:6443/version\": tls: failed to verify certificate: x509: certificate signed by unknown authority"
    ],
    "stdout": "",
    "stdout_lines": []
}

verbose output show the proxy env

<10.XX.XX.XX> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o 'IdentityFile="/home/user/.ssh/user-ansible"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="user"' -o ConnectTimeout=10 -o 'ControlPath="/home/user/.ansible/cp/2d48a40eb9"'10.XX.XX.XX '/bin/sh -c '"'"'sudo -H -S -n  -u root /bin/sh -c '"'"'"'"'"'"'"'"'echo BECOME-SUCCESS-hrfrnpvqtsxvhxtnqdnfjewoftnqwsdo ; ALL_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' FTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTPS_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' HTTP_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' NO_PROXY='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' all_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' ftp_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' http_proxy=http://foobar https_proxy=http://foobar no_proxy='"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"''"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"'"' /usr/bin/python3'"'"'"'"'"'"'"'"' && sleep 0'"'"''
Escalation succeeded
<10.XX.XX.XX> (1, b'\n{"stdout": "", "stderr": "Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "command": "/usr/local/bin/helm list --output=yaml --filter kubelet-csr-approver", "failed": true, "msg": "Failure when executing Helm command. Exited 1.\\nstdout: \\nstderr: Error: Kubernetes cluster unreachable: Get \\"https://lb-apiserver.kubernetes.local:6443/version\\": tls: failed to verify certificate: x509: certificate signed by unknown authority\\n", "invocation": {"module_args": {"atomic": true, "binary_path": "/usr/local/bin/helm", "name": "kubelet-csr-approver", "namespace": "kube-system", "chart_ref": "kubelet-csr-approver/kubelet-csr-approver", "chart_version": "0.2.8", "wait": true, "values": {}, "release_name": "kubelet-csr-approver", "release_namespace": "kube-system", "release_values": {}, "validate_certs": true, "dependency_update": false, "release_state": "present", "values_files": [], "update_repo_cache": false, "disable_hook": false, "force": false, "purge": true, "create_namespace": false, "replace": false, "skip_crds": false, "context": null, "kubeconfig": null, "host": null, "ca_cert": null, "api_key": null, "chart_repo_url": null, "wait_timeout": null, "timeout": null, "post_renderer": null, "history_max": null, "set_values": null}}}\n', b"OpenSSH_8.9p1 Ubuntu-3ubuntu0.6, OpenSSL 3.0.2 15 Mar 2022\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 10.XX.XX.XX is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/user/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/user/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 418983\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet: read header failed: Broken pipe\r\ndebug2: Received exit status from master 1\r\n")

Anything else we need to know

My solution is adding back no_proxy env (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/kubernetes-apps/kubelet-csr-approver/meta/main.yml)

---
dependencies:
  - role: helm-apps
    when:
      - inventory_hostname == groups['kube_control_plane'][0]
      - kubelet_csr_approver_enabled
    environment:
      http_proxy: "{{ http_proxy | default('') }}"
      https_proxy: "{{ https_proxy | default('') }}"
      no_proxy: "{{ no_proxy | default('') }}" # <-------  add this
    release_common_opts: {}
    releases:
      - name: kubelet-csr-approver
        namespace: "{{ kubelet_csr_approver_namespace }}"
        chart_ref: "{{ kubelet_csr_approver_chart_ref }}"
        chart_version: "{{ kubelet_csr_approver_chart_version }}"
        wait: true
        values: "{{ kubelet_csr_approver_values }}"
    repositories:
      - name: "{{ kubelet_csr_approver_repository_name }}"
        url: "{{ kubelet_csr_approver_repository_url }}"

I also found a similar structure at custom_cni (https://github.com/kubernetes-sigs/kubespray/blob/v2.24.1/roles/network_plugin/custom_cni/meta/main.yml)

[KubeKyrie]

Could u show the kube-vip setting?

[andrewfung729]

Thanks for your reply. For your information, after I added back the no_proxy env on playbook, the issue no longer occurred again.

## `inventory/mycluster/group_vars/k8s_cluster/k8s-cluster.yml`
kube_proxy_strict_arp: true

## `inventory/mycluster/group_vars/k8s_cluster/addons.yml`
# Kube VIP
kube_vip_enabled: true
kube_vip_arp_enabled: true
kube_vip_controlplane_enabled: true
kube_vip_address: 10.XX.XX.XX
loadbalancer_apiserver:
  address: "{{ kube_vip_address }}"
  port: 6443
kube_vip_interface: enp6s19
kube_vip_lb_enable: true
kube_vip_services_enabled: false
kube_vip_enableServicesElection: true

[andrewfung729]

@KubeKyrie I think it just needs a simple fix no_proxy: "{{ no_proxy | default('') }}". Can I make a pull request for that?

FYI, to enable kubelet-csr-approver we need kubelet_rotate_server_certificates: true.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[andrewfung729]

/remove-lifecycle stale