kubekey
kubekey copied to clipboard
kubesphere
Offline installation failed
What is version of KubeKey has the issue?
v3.0.10
What is your os environment?
centos7.9
KubeKey config file
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: ks-master-1, address: 192.168.31.202, internalAddress: 192.168.31.202, user: root, password: "QWEqwe@123"}
roleGroups:
etcd:
- ks-master-1
control-plane:
- ks-master-1
worker:
- ks-master-1
registry:
- ks-master-1
controlPlaneEndpoint:
## Internal loadbalancer for apiservers
# internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.25.10
clusterName: cluster.local
autoRenewCerts: true
containerManager: containerd
etcd:
type: kubekey
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
multusCNI:
enabled: false
registry:
type: harbor
auths:
"dockerhub.kubekey.local":
username: admin
password: Harbor12345
certsPath: "/etc/docker/certs.d/dockerhub.kubekey.local"
privateRegistry: "dockerhub.kubekey.local"
namespaceOverride: "kubesphereio"
registryMirrors: []
insecureRegistries: []
addons: []
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.4.0
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
zone: ""
local_registry: ""
namespace_override: ""
# dev_tag: ""
etcd:
monitoring: true
endpointIps: localhost
port: 2379
tlsEnable: true
common:
core:
console:
enableMultiLogin: true
port: 30880
type: NodePort
# apiserver:
# resources: {}
# controllerManager:
# resources: {}
redis:
enabled: false
enableHA: false
volumeSize: 2Gi
openldap:
enabled: false
volumeSize: 2Gi
minio:
volumeSize: 20Gi
monitoring:
# type: external
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
GPUMonitoring:
enabled: false
gpu:
kinds:
- resourceName: "nvidia.com/gpu"
resourceType: "GPU"
default: true
es:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: false
username: ""
password: ""
externalElasticsearchHost: ""
externalElasticsearchPort: ""
opensearch:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
enabled: true
logMaxAge: 7
opensearchPrefix: whizard
basicAuth:
enabled: true
username: "admin"
password: "admin"
externalOpensearchHost: ""
externalOpensearchPort: ""
dashboard:
enabled: false
alerting:
enabled: true
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: true
# operator:
# resources: {}
# webhook:
# resources: {}
devops:
enabled: true
jenkinsCpuReq: 0.5
jenkinsCpuLim: 1
jenkinsMemoryReq: 2Gi
jenkinsMemoryLim: 2Gi
jenkinsVolumeSize: 16Gi
events:
enabled: true
# operator:
# resources: {}
# exporter:
# resources: {}
# ruler:
# enabled: true
# replicas: 2
# resources: {}
logging:
enabled: true
logsidecar:
enabled: true
replicas: 2
# resources: {}
metrics_server:
enabled: true
monitoring:
storageClass: ""
node_exporter:
port: 9100
# resources: {}
# kube_rbac_proxy:
# resources: {}
# kube_state_metrics:
# resources: {}
# prometheus:
# replicas: 1
# volumeSize: 20Gi
# resources: {}
# operator:
# resources: {}
# alertmanager:
# replicas: 1
# resources: {}
# notification_manager:
# resources: {}
# operator:
# resources: {}
# proxy:
# resources: {}
gpu:
nvidia_dcgm_exporter:
enabled: false
# resources: {}
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: true
ippool:
type: calico
topology:
type: weave-scope
openpitrix:
store:
enabled: true
servicemesh:
enabled: true
istio:
components:
ingressGateways:
- name: istio-ingressgateway
enabled: false
cni:
enabled: false
edgeruntime:
enabled: false
kubeedge:
enabled: false
cloudCore:
cloudHub:
advertiseAddress:
- ""
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
# resources: {}
# hostNetWork: false
iptables-manager:
enabled: true
mode: "external"
# resources: {}
# edgeService:
# resources: {}
gatekeeper:
enabled: false
# controller_manager:
# resources: {}
# audit:
# resources: {}
terminal:
timeout: 600
A clear and concise description of what happend.
Offline installation failed：：get manifest list failed by module cache
./kk create cluster -f config-sample.yaml-a kubesphere.tar.gz --with-packages
Relevant log output
19:30:06 CST success: [ks-master-1]
19:30:06 CST [ConfigureOSModule] configure the ntp server for each node
19:30:06 CST skipped: [ks-master-1]
19:30:06 CST [KubernetesStatusModule] Get kubernetes cluster status
19:30:07 CST success: [ks-master-1]
19:30:07 CST [InstallContainerModule] Sync containerd binaries
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [InstallContainerModule] Sync crictl binaries
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [InstallContainerModule] Generate containerd service
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [InstallContainerModule] Generate containerd config
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [InstallContainerModule] Generate crictl config
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [InstallContainerModule] Enable containerd
19:30:07 CST skipped: [ks-master-1]
19:30:07 CST [CopyImagesToRegistryModule] Copy images to a private registry from an artifact OCI Path
19:30:07 CST Source: oci:/root/k8s_1_25_10phere3_4_0/kubekey/images:calico:cni:v3.23.2-amd64
19:30:07 CST Destination: docker://dockerhub.kubekey.local/kubesphereio/cni:v3.23.2-amd64
Getting image source signatures
Getting image source signatures
Getting image source signatures
Getting image source signatures
Getting image source signatures
19:30:07 CST success: [LocalHost]
19:30:07 CST [CopyImagesToRegistryModule] Push multi-arch manifest to private registry
19:30:07 CST message: [LocalHost]
get manifest list failed by module cache
19:30:07 CST failed: [LocalHost]
error: Pipeline[CreateClusterPipeline] execute failed: Module[CopyImagesToRegistryModule] exec failed:
failed: [LocalHost] [PushManifest] exec failed after 1 retries: get manifest list failed by module cache
Additional information
I have completed the Harbor installation locally
I've run into the same problem with kubekey v3.0.10。I've never learned go before, which made it hard for me to read the source code and locate the problem.
I think I might find out the cause by chance. I noticed kk was downloading cni v1.2.0 while creating cluster. But the version specified in manifest.yaml is v0.9.1. I changed the version of cni and tried again. The images were pushed into harbor successfully. But I ran into a new problem:
16:08:11 CST [PullModule] Start to pull images on all nodes 16:08:11 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:11 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:11 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:11 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:11.772601 5657 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:11 CST retry: [node3] 16:08:11 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:11.823472 5611 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:11 CST retry: [node2] 16:08:12 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:12.687828 13391 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:12 CST retry: [node1] 16:08:16 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:16 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:16 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:16.859918 5677 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:16 CST retry: [node3] 16:08:16 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:16.882201 5629 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:16 CST retry: [node2] 16:08:17 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:17 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:17.748254 13409 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:17 CST retry: [node1] 16:08:21 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:21 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:21 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:21.914042 5695 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:21 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:21.941021 5647 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:22 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.8 16:08:22 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:22.793405 13496 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 16:08:22 CST failed: [node3] 16:08:22 CST failed: [node2] 16:08:22 CST failed: [node1] error: Pipeline[CreateClusterPipeline] execute failed: Module[PullModule] exec failed: failed: [node3] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:21.914042 5695 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 failed: [node2] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:21.941021 5647 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1 failed: [node1] [PullImages] exec failed after 3 retries: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH crictl pull dockerhub.kubekey.local/kubesphereio/pause:3.8 --platform amd64" E0816 16:08:22.793405 13496 remote_image.go:238] "PullImage from image service failed" err="rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found" image="dockerhub.kubekey.local/kubesphereio/pause:3.8" FATA[0000] pulling image: rpc error: code = NotFound desc = failed to pull and unpack image "dockerhub.kubekey.local/kubesphereio/pause:3.8": failed to resolve reference "dockerhub.kubekey.local/kubesphereio/pause:3.8": dockerhub.kubekey.local/kubesphereio/pause:3.8: not found: Process exited with status 1
It seems manifest.yaml missed out pause:3.8. I'll try again
Just forget to mention I changed the version of kubernetes to v1.27.2.
hello hello Did you solve this problem? I have the same problem, can you tell me how to solve it
hello! Did you solve this problem? I have the same problem, could you tell me how to solve it?thanks
hello! Did you solve this problem? I have the same problem, could you tell me how to solve it?thanks
