kubekey icon indicating copy to clipboard operation
kubekey copied to clipboard

Published 20 hours ago • verified publisher icon kubesphere

air-gapped environment install failed

Open Thatwho opened this issue 4 months ago • 0 comments

What is version of KubeKey has the issue?

v3.1.6

What is your os environment?

Ubuntu 22.04

KubeKey config file

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: node1, address: 192.168.120.3, internalAddress: 192.168.120.3, user: user, password: "123456"}
  roleGroups:
    etcd:
    - node1
    control-plane: 
    - node1
    worker:
    - node1
    registry:
    - node1
  controlPlaneEndpoint:
    ## Internal loadbalancer for apiservers 
    # internalLoadbalancer: haproxy

    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.28.13
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    ## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
    multusCNI:
      enabled: false
  registry:
    type: harbor
    auths:
      "reg.kubekey.local":
        username: admin
        password: Harbor12345
        certsPath: /etc/docker/certs.d/reg.kubekey.local
    privateRegistry: "reg.kubekey.local"
    namespaceOverride: "kubesphere"
    registryMirrors: []
    insecureRegistries: []
  addons: []

A clear and concise description of what happend.

I'm facing an issue when installing Kubernetes in an air-gapped environment using Kubekey. The installation fails, and when I check the logs using journalctl -xfeu kubelet, I see the following error messages:

Oct 19 13:13:54 node1 kubelet[647]: E1019 13:13:54.628298     647 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-scheduler-node1.17ffcfa08ae2e7f3", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-scheduler-node1", UID:"6a9443ee7c86d00dc2d503017a48ce21", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"FailedCreatePodSandBox", Message:"Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable", Source:v1.EventSource{Component:"kubelet", Host:"node1"}, FirstTimestamp:time.Date(2024, time.October, 19, 9, 11, 10, 365403123, time.Local), LastTimestamp:time.Date(2024, time.October, 19, 9, 15, 52, 951162615, time.Local), Count:22, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"kubelet", ReportingInstance:"node1"}': 'Patch "https://lb.kubesphere.local:6443/api/v1/namespaces/kube-system/events/kube-scheduler-node1.17ffcfa08ae2e7f3": dial tcp 192.168.120.3:6443: connect: connection refused'(may retry after sleeping)
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.185117     647 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://lb.kubesphere.local:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node1?timeout=10s\": dial tcp 192.168.120.3:6443: connect: connection refused" interval="7s"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983785     647 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983905     647 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-scheduler-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983933     647 kuberuntime_manager.go:1181] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-scheduler-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983991     647 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-scheduler-node1_kube-system(6a9443ee7c86d00dc2d503017a48ce21)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-scheduler-node1_kube-system(6a9443ee7c86d00dc2d503017a48ce21)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"registry.k8s.io/pause:3.8\\\": failed to pull image \\\"registry.k8s.io/pause:3.8\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.8\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.8\\\": failed to do request: Head \\\"https://registry.k8s.io/v2/pause/manifests/3.8\\\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable\"" pod="kube-system/kube-scheduler-node1" podUID="6a9443ee7c86d00dc2d503017a48ce21"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986432     647 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986460     647 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-controller-manager-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986475     647 kuberuntime_manager.go:1181] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-controller-manager-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986508     647 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-controller-manager-node1_kube-system(8e63188c7b866f30b73a171d356edc93)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-controller-manager-node1_kube-system(8e63188c7b866f30b73a171d356edc93)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"registry.k8s.io/pause:3.8\\\": failed to pull image \\\"registry.k8s.io/pause:3.8\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.8\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.8\\\": failed to do request: Head \\\"https://registry.k8s.io/v2/pause/manifests/3.8\\\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable\"" pod="kube-system/kube-controller-manager-node1" podUID="8e63188c7b866f30b73a171d356edc93"

It seems like the kubelet is trying to pull the pause image from the internet, but since this is an air-gapped environment, the network is unreachable.

Did I miss any steps during the air-gapped setup that could have caused this? How should I proceed to ensure the kubelet pulls the required images from the local repository instead of the internet?

Relevant log output

Oct 19 13:13:54 node1 kubelet[647]: E1019 13:13:54.628298     647 event.go:289] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"kube-scheduler-node1.17ffcfa08ae2e7f3", GenerateName:"", Namespace:"kube-system", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Pod", Namespace:"kube-system", Name:"kube-scheduler-node1", UID:"6a9443ee7c86d00dc2d503017a48ce21", APIVersion:"v1", ResourceVersion:"", FieldPath:""}, Reason:"FailedCreatePodSandBox", Message:"Failed to create pod sandbox: rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable", Source:v1.EventSource{Component:"kubelet", Host:"node1"}, FirstTimestamp:time.Date(2024, time.October, 19, 9, 11, 10, 365403123, time.Local), LastTimestamp:time.Date(2024, time.October, 19, 9, 15, 52, 951162615, time.Local), Count:22, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"kubelet", ReportingInstance:"node1"}': 'Patch "https://lb.kubesphere.local:6443/api/v1/namespaces/kube-system/events/kube-scheduler-node1.17ffcfa08ae2e7f3": dial tcp 192.168.120.3:6443: connect: connection refused'(may retry after sleeping)
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.185117     647 controller.go:146] "Failed to ensure lease exists, will retry" err="Get \"https://lb.kubesphere.local:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/node1?timeout=10s\": dial tcp 192.168.120.3:6443: connect: connection refused" interval="7s"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983785     647 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983905     647 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-scheduler-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983933     647 kuberuntime_manager.go:1181] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-scheduler-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.983991     647 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-scheduler-node1_kube-system(6a9443ee7c86d00dc2d503017a48ce21)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-scheduler-node1_kube-system(6a9443ee7c86d00dc2d503017a48ce21)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"registry.k8s.io/pause:3.8\\\": failed to pull image \\\"registry.k8s.io/pause:3.8\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.8\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.8\\\": failed to do request: Head \\\"https://registry.k8s.io/v2/pause/manifests/3.8\\\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable\"" pod="kube-system/kube-scheduler-node1" podUID="6a9443ee7c86d00dc2d503017a48ce21"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986432     647 remote_runtime.go:193] "RunPodSandbox from runtime service failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986460     647 kuberuntime_sandbox.go:72] "Failed to create sandbox for pod" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-controller-manager-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986475     647 kuberuntime_manager.go:1181] "CreatePodSandbox for pod failed" err="rpc error: code = Unknown desc = failed to get sandbox image \"registry.k8s.io/pause:3.8\": failed to pull image \"registry.k8s.io/pause:3.8\": failed to pull and unpack image \"registry.k8s.io/pause:3.8\": failed to resolve reference \"registry.k8s.io/pause:3.8\": failed to do request: Head \"https://registry.k8s.io/v2/pause/manifests/3.8\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable" pod="kube-system/kube-controller-manager-node1"
Oct 19 13:13:55 node1 kubelet[647]: E1019 13:13:55.986508     647 pod_workers.go:1300] "Error syncing pod, skipping" err="failed to \"CreatePodSandbox\" for \"kube-controller-manager-node1_kube-system(8e63188c7b866f30b73a171d356edc93)\" with CreatePodSandboxError: \"Failed to create sandbox for pod \\\"kube-controller-manager-node1_kube-system(8e63188c7b866f30b73a171d356edc93)\\\": rpc error: code = Unknown desc = failed to get sandbox image \\\"registry.k8s.io/pause:3.8\\\": failed to pull image \\\"registry.k8s.io/pause:3.8\\\": failed to pull and unpack image \\\"registry.k8s.io/pause:3.8\\\": failed to resolve reference \\\"registry.k8s.io/pause:3.8\\\": failed to do request: Head \\\"https://registry.k8s.io/v2/pause/manifests/3.8\\\": dial tcp: lookup registry.k8s.io on 114.114.114.114:53: dial udp 114.114.114.114:53: connect: network is unreachable\"" pod="kube-system/kube-controller-manager-node1" podUID="8e63188c7b866f30b73a171d356edc93"

Additional information

No response

Thatwho avatar Oct 19 '24 13:10 Thatwho