kubectl API server authentication error when connecting through apiserver load balancer

[khatrig]

Environment:

• Cloud provider or hardware configuration: On-prem baremetal nodes

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"): Ubuntu 20.04

• Version of Ansible (ansible --version): 2.10.11

• Version of Python (python --version): 3.8

Kubespray version (commit) (git rev-parse --short HEAD): release-2.17

Network plugin used: calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

Command used to invoke ansible:

Output of ansible run:

Anything else do we need to know:

ENV: I have a ha k8s 1.21 cluster setup using kubespray + keepalived & nginx for api load balancing. I'm getting below intermittent error when connecting to the cluster using external LB

The issue: $kubectl get nodes Unable to connect to the server: x509: certificate signed by unknown authority

Root cause: This is happening because all 3 apiservers have different apiserver.crt. kubeconfig has field certificate-authority-data which is encoded apiserver.crt from one of the nodes. when the load balancer sends the request to any other apiserver, the request fails. Earlier kubespray used to have the same apiserver certificates on all control plane nodes. This used to work as expected before PR.

Fix: I was able to fix the issue by copying the same apiserver.{crt,key} to all 3 control plane nodes.

[cristicalin]

/cc @champtar

I'm not sure what would be the proper fix here, on the one hand generating per node certificates makes it cleaner but it also causes issues when kubeconfig is storing the certificate-authority-data.

[khatrig]

Can we go back to the older way of creating apiserver crt/key on one control plane node and copying it on the rest of the nodes?

I'm using below as a workaround for now:

---

• name: Fetch API Server certs on ansible host fetch: src: /etc/kubernetes/ssl/{{ item }} dest: /tmp/apiserver/ flat: yes loop:

  • apiserver.crt
  • apiserver.key when: inventory_hostname == groups['kube_control_plane'][0] tags: apiserver

• name: Copy same API Server key to all control plane nodes copy: src: /tmp/apiserver/{{ item }} dest: /etc/kubernetes/ssl/ loop:

  • apiserver.crt
  • apiserver.key when: inventory_hostname in groups['kube_control_plane']

Let me know if this helps. I can probably raise a PR with this.

Thanks.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[khatrig]

/remove-lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue.

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue or PR with /reopen
• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.