kubekey
kubekey copied to clipboard
Published
20 hours ago •
kubesphere
kubesphere
how to save image to my own harbor registry
What is version of KubeKey has the issue?
[root@harbor kubesphere]# kk version version.BuildInfo{Version:"2.0.0-rc.4", GitCommit:"a337b297", GitTreeState:"", GoVersion:"go1.17.7"}
What is your os environment?
centos7
KubeKey config file
apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
name: sample
spec:
hosts:
- {name: node1, address: 192.168.93.60, internalAddress: 192.168.93.60, user: root, password: "123456"}
- {name: node2, address: 192.168.93.61, internalAddress: 192.168.93.61, user: root, password: "123456"}
- {name: node3, address: 192.168.93.62, internalAddress: 192.168.93.62, user: root, password: "123456"}
roleGroups:
etcd:
- node1
control-plane:
- node1
worker:
- node2
- node3
controlPlaneEndpoint:
## Internal loadbalancer for apiservers
# internalLoadbalancer: haproxy
domain: lb.kubesphere.local
address: ""
port: 6443
kubernetes:
version: v1.21.5
clusterName: cluster.local
containerManager: containerd
network:
plugin: calico
kubePodsCIDR: 10.233.64.0/18
kubeServiceCIDR: 10.233.0.0/18
## multus support. https://github.com/k8snetworkplumbingwg/multus-cni
multusCNI:
enabled: false
registry:
type: harbor
auths:
"dockerhub.kubekey.local":
username: admin
password: Harbor12345
plainHTTP: yes
privateRegistry: "dockerhub.kubekey.local"
namespaceOverride: "kubesphereio"
registryMirrors: []
insecureRegistries: ["dockerhub.kubekey.local"]
addons: []
---
apiVersion: installer.kubesphere.io/v1alpha1
kind: ClusterConfiguration
metadata:
name: ks-installer
namespace: kubesphere-system
labels:
version: v3.2.1
spec:
persistence:
storageClass: ""
authentication:
jwtSecret: ""
local_registry: ""
namespace_override: ""
# dev_tag: ""
etcd:
monitoring: true
endpointIps: localhost
port: 2379
tlsEnable: true
common:
core:
console:
enableMultiLogin: true
port: 30880
type: NodePort
# apiserver:
# resources: {}
# controllerManager:
# resources: {}
redis:
enabled: true
volumeSize: 2Gi
openldap:
enabled: true
volumeSize: 2Gi
minio:
volumeSize: 5Gi
monitoring:
# type: external
endpoint: http://prometheus-operated.kubesphere-monitoring-system.svc:9090
GPUMonitoring:
enabled: false
gpu:
kinds:
- resourceName: "nvidia.com/gpu"
resourceType: "GPU"
default: true
es:
# master:
# volumeSize: 4Gi
# replicas: 1
# resources: {}
# data:
# volumeSize: 20Gi
# replicas: 1
# resources: {}
logMaxAge: 7
elkPrefix: logstash
basicAuth:
enabled: true
username: "admin"
password: "Ks@123456"
externalElasticsearchHost: ""
externalElasticsearchPort: ""
alerting:
enabled: true
# thanosruler:
# replicas: 1
# resources: {}
auditing:
enabled: true
# operator:
# resources: {}
# webhook:
# resources: {}
devops:
enabled: true
jenkinsMemoryLim: 2Gi
jenkinsMemoryReq: 1500Mi
jenkinsVolumeSize: 8Gi
jenkinsJavaOpts_Xms: 512m
jenkinsJavaOpts_Xmx: 512m
jenkinsJavaOpts_MaxRAM: 2g
events:
enabled: true
# operator:
# resources: {}
# exporter:
# resources: {}
# ruler:
# enabled: true
# replicas: 2
# resources: {}
logging:
enabled: true
containerruntime: containerd
logsidecar:
enabled: true
replicas: 2
# resources: {}
metrics_server:
enabled: false
monitoring:
storageClass: ""
# kube_rbac_proxy:
# resources: {}
# kube_state_metrics:
# resources: {}
# prometheus:
# replicas: 1
# volumeSize: 20Gi
# resources: {}
# operator:
# resources: {}
# adapter:
# resources: {}
# node_exporter:
# resources: {}
# alertmanager:
# replicas: 1
# resources: {}
# notification_manager:
# resources: {}
# operator:
# resources: {}
# proxy:
# resources: {}
gpu:
nvidia_dcgm_exporter:
enabled: false
# resources: {}
multicluster:
clusterRole: none
network:
networkpolicy:
enabled: false
ippool:
type: none
topology:
type: none
openpitrix:
store:
enabled: true
servicemesh:
enabled: true
kubeedge:
enabled: false
cloudCore:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
cloudhubPort: "10000"
cloudhubQuicPort: "10001"
cloudhubHttpsPort: "10002"
cloudstreamPort: "10003"
tunnelPort: "10004"
cloudHub:
advertiseAddress:
- ""
nodeLimit: "100"
service:
cloudhubNodePort: "30000"
cloudhubQuicNodePort: "30001"
cloudhubHttpsNodePort: "30002"
cloudstreamNodePort: "30003"
tunnelNodePort: "30004"
edgeWatcher:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
edgeWatcherAgent:
nodeSelector: {"node-role.kubernetes.io/worker": ""}
tolerations: []
A clear and concise description of what happend.
https://github.com/kubesphere/kubekey/blob/master/docs/zh/manifest_and_artifact.md
kk create cluster -f config-sample.yaml -a kubesphere.tar.gz --with-kubernetes v1.21.5 --with-kubesphere v3.2.1 --with-packages
Relevant log output
err
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
failed: [node3] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
failed: [node1] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
Additional information
No response
If you use the harbor registry, you need to create the harbor project manually. You can refer to this doc: https://mp.weixin.qq.com/s/hjtNfSRVYH1O2o_dj6ET4A
yes i see this ,but when it create harbor project ,then install cluster ,but where are the step of push image to harbor registry?
this command not push image to harbor registry ,just to pull from harbor ,but my harbor is empty , just have one empty project
kk create cluster -f config-sample.yaml --with-kubernetes v1.21.5 --with-kubesphere v3.2.1
so it will error when i run this command
18:09:34 CST skipped: [node2]
18:09:34 CST skipped: [node1]
18:09:34 CST skipped: [node3]
18:09:34 CST [InstallContainerModule] Enable docker
18:09:34 CST skipped: [node1]
18:09:34 CST skipped: [node3]
18:09:34 CST skipped: [node2]
18:09:34 CST [InstallContainerModule] Add auths to container runtime
18:09:37 CST success: [node3]
18:09:37 CST success: [node2]
18:09:37 CST success: [node1]
18:09:37 CST [PullModule] Start to pull images on all nodes
18:09:37 CST message: [node1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:37 CST message: [node3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:37 CST message: [node2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:37 CST message: [node2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:37 CST retry: [node2]
18:09:37 CST message: [node1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:37 CST retry: [node1]
18:09:37 CST message: [node3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:37 CST retry: [node3]
18:09:42 CST message: [node2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:42 CST message: [node1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:42 CST message: [node3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:42 CST message: [node2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:42 CST retry: [node2]
18:09:42 CST message: [node1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:42 CST retry: [node1]
18:09:42 CST message: [node3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:42 CST retry: [node3]
18:09:47 CST message: [node2]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:47 CST message: [node1]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:47 CST message: [node3]
downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1
18:09:47 CST message: [node2]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:48 CST message: [node3]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:48 CST message: [node1]
pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
18:09:48 CST failed: [node2]
18:09:48 CST failed: [node3]
18:09:48 CST failed: [node1]
error: Pipeline[CreateClusterPipeline] execute failed: Module[PullModule] exec failed:
failed: [node2] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
failed: [node3] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
failed: [node1] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1"
Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1
[root@harbor kubesphere]# ^C
[root@harbor kubesphere]#
Please see https://github.com/kubesphere/kubekey/pull/1094
And i will add some doc for this command later.
yes ,i have config it , you mean it will push by default, but it skip
registry:
type: harbor
auths:
"dockerhub.kubekey.local":
username: admin
password: Harbor12345
plainHTTP: yes
privateRegistry: "dockerhub.kubekey.local"
namespaceOverride: "kubesphereio"
registryMirrors: []
insecureRegistries: ["dockerhub.kubekey.local"]
addons: []
this command not push image to harbor registry ,just to pull from harbor ,but my harbor is empty , just have one empty project
kk create cluster -f config-sample.yaml --with-kubernetes v1.21.5 --with-kubesphere v3.2.1so it will error when i run this command
18:09:34 CST skipped: [node2] 18:09:34 CST skipped: [node1] 18:09:34 CST skipped: [node3] 18:09:34 CST [InstallContainerModule] Enable docker 18:09:34 CST skipped: [node1] 18:09:34 CST skipped: [node3] 18:09:34 CST skipped: [node2] 18:09:34 CST [InstallContainerModule] Add auths to container runtime 18:09:37 CST success: [node3] 18:09:37 CST success: [node2] 18:09:37 CST success: [node1] 18:09:37 CST [PullModule] Start to pull images on all nodes 18:09:37 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:37 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:37 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:37 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:37 CST retry: [node2] 18:09:37 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:37 CST retry: [node1] 18:09:37 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:37 CST retry: [node3] 18:09:42 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:42 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:42 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:42 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:42 CST retry: [node2] 18:09:42 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:42 CST retry: [node1] 18:09:42 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:42 CST retry: [node3] 18:09:47 CST message: [node2] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:47 CST message: [node1] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:47 CST message: [node3] downloading image: dockerhub.kubekey.local/kubesphereio/pause:3.4.1 18:09:47 CST message: [node2] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:48 CST message: [node3] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:48 CST message: [node1] pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 18:09:48 CST failed: [node2] 18:09:48 CST failed: [node3] 18:09:48 CST failed: [node1] error: Pipeline[CreateClusterPipeline] execute failed: Module[PullModule] exec failed: failed: [node2] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 failed: [node3] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 failed: [node1] [PullImages] exec failed after 3 retires: pull image failed: Failed to exec command: sudo -E /bin/bash -c "env PATH=$PATH docker pull dockerhub.kubekey.local/kubesphereio/pause:3.4.1" Error response from daemon: unknown: repository kubesphereio/pause not found: Process exited with status 1 [root@harbor kubesphere]# ^C [root@harbor kubesphere]#
You should use the command with -a option, otherwise the pipeline will not include the push module.
kk create cluster -a kubekey.tar.gz
yes , but i have new error when i run kk artifact images push -f config-sample.yaml
push dockerhub.kubekey.local/kubesphereio/openpitrix-jobs:v3.2.1 success
push dockerhub.kubekey.local/kubesphereio/pause:3.4.1 success
push dockerhub.kubekey.local/kubesphereio/pause:3.5 success
push dockerhub.kubekey.local/kubesphereio/perl:latest success
push dockerhub.kubekey.local/kubesphereio/pilot:1.11.1 success
push dockerhub.kubekey.local/kubesphereio/pod2daemon-flexvol:v3.20.0 success
push dockerhub.kubekey.local/kubesphereio/prometheus-config-reloader:v0.43.2 success
push dockerhub.kubekey.local/kubesphereio/prometheus-operator:v0.43.2 success
push dockerhub.kubekey.local/kubesphereio/prometheus:v2.26.0 success
push dockerhub.kubekey.local/kubesphereio/provisioner-localpv:2.10.1 success
18:50:49 CST message: [LocalHost]
import image registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1 failed: ctr: failed to ingest "blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058": failed commit on ref "tar-blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058": commit failed: "tar-blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058" failed size validation: 22708224 != 28567944: failed precondition
: exit status 1
18:50:49 CST failed: [LocalHost]
error: Pipeline[ArtifactImagesPushPipeline] execute failed: Module[PushModule] exec failed:
failed: [LocalHost] [PushImages] exec failed after 1 retires: import image registry.cn-beijing.aliyuncs.com/kubesphereio/proxyv2:1.11.1 failed: ctr: failed to ingest "blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058": failed commit on ref "tar-blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058": commit failed: "tar-blobs/sha256/16ec32c2132b43494832a05f2b02f7a822479f8250c173d0ab27b3de78b2f058" failed size validation: 22708224 != 28567944: failed precondition
: exit status 1
There are something wrong in this image file. You may need to push this image manually.
i lost so many options to do
add this and create harbor projects, the official docs have been say that.
local_registry: ""
``