kubekey icon indicating copy to clipboard operation
kubekey copied to clipboard

Published 20 hours ago • verified publisher icon kubesphere

reinstall failed when Offline install interruption

Open willzhang opened this issue 2 years ago • 4 comments

What is version of KubeKey has the issue?

2.2.2

What is your os environment?

ubuntu 22.04

KubeKey config file

root@ubuntu:/data/kubesphere/v3.3.0# cat config-sample.yaml

apiVersion: kubekey.kubesphere.io/v1alpha2
kind: Cluster
metadata:
  name: sample
spec:
  hosts:
  - {name: node1, address: 192.168.72.40, internalAddress: 192.168.72.40, user: root, password: "123456"}
  - {name: node2, address: 192.168.72.41, internalAddress: 192.168.72.41, user: root, password: "123456"}
  - {name: node3, address: 192.168.72.42, internalAddress: 192.168.72.42, user: root, password: "123456"}
  - {name: harbor, address: 192.168.72.43, internalAddress: 192.168.72.43, user: root, password: "123456"}
  roleGroups:
    etcd:
    - node1
    control-plane: 
    - node1
    worker:
    - node1
    - node2
    - node3
    registry:
    - harbor
  controlPlaneEndpoint:
    domain: lb.kubesphere.local
    address: ""
    port: 6443
  kubernetes:
    version: v1.21.5
    clusterName: cluster.local
    autoRenewCerts: true
    containerManager: containerd
  etcd:
    type: kubekey
  network:
    plugin: calico
    kubePodsCIDR: 10.233.64.0/18
    kubeServiceCIDR: 10.233.0.0/18
    multusCNI:
      enabled: false
  registry:
    type: harbor
    auths:
      "dockerhub.kubekey.local":
        username: admin
        password: Harbor12345
    privateRegistry: "dockerhub.kubekey.local"
    namespaceOverride: "kubesphereio"
    registryMirrors: []
    insecureRegistries: []
  addons: []

A clear and concise description of what happend.

1、run kk create cluster -f config-sample.yaml -a kubernetes-v1.21.5-artifact.tar.gz --with-packages the install process will always stop here i must interruption by hand

Preparing to unpack .../3-ipset_7.15-1build1_amd64.deb ...
Unpacking ipset (7.15-1build1) ...............] 
Selecting previously unselected package ipvsadm.
Preparing to unpack .../4-ipvsadm_1.31-1build2_amd64.deb ...
Unpacking ipvsadm (1:1.31-1build2) ...........] 
Selecting previously unselected package socat.] 
Preparing to unpack .../5-socat_1.7.4.1-3ubuntu4_amd64.deb ...
Unpacking socat (1.7.4.1-3ubuntu4) ...........] 
Setting up ipvsadm (1:1.31-1build2) ..........] 
Setting up conntrack (1:1.4.6-2build2) .......] 
Setting up ebtables (2.0.11-4build2) .........] 
Setting up socat (1.7.4.1-3ubuntu4) ..........] 
Setting up libipset13:amd64 (7.15-1build1) ...] 
Setting up ipset (7.15-1build1) ...#######....] 
Processing triggers for man-db (2.10.2-1) ....] 
Processing triggers for libc-bin (2.35-0ubuntu3) ...

Scanning processes...                             
Scanning linux images...                          

Running kernel seems to be up-to-date.

No services need to be restarted.

No containers need to be restarted.

No user sessions are running outdated binaries.

No VM guests are running outdated hypervisor
 (qemu) binaries on this host.


^C
root@ubuntu:/data/kubesphere/v3.3.0# 
root@ubuntu:/data/kubesphere/v3.3.0#

2、when i reinstall with kk create cluster -f config-sample.yaml -a kubernetes-v1.21.5-artifact.tar.gz --with-packages , here will have two problems.

Relevant log output

problems 1



rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libx11/libx11-data_1.7.5-1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libxau/libxau6_1.0.9-1build5_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libxcb/libxcb1_1.14-3ubuntu3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libxdmcp/libxdmcp6_1.1.3-0ubuntu5_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libxext/libxext6_1.3.4-1build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/libx/libxmu/libxmuu1_1.1.3-3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/ncurses/ncurses-term_6.3-2_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/ndctl/libdaxctl1_72.1-1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/ndctl/libndctl6_72.1-1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/netbase/netbase_6.3_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/netkit-telnet/telnet_0.17-44build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/net-snmp/libsnmp40_5.9.1+dfsg-1ubuntu2.1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/net-snmp/libsnmp-base_5.9.1+dfsg-1ubuntu2.1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/net-tools/net-tools_1.60+git20181103.0eebece-1ubuntu5_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/nfs-utils/libnfsidmap1_2.6.1-1ubuntu1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/nfs-utils/nfs-common_2.6.1-1ubuntu1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/n/numactl/libnuma1_2.0.14-3ubuntu2_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/o/oath-toolkit/liboath0_2.6.7-3build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/o/openssh/openssh-client_8.9p1-3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/o/openssh/openssh-sftp-server_8.9p1-3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/o/openssh/openssh-server_8.9p1-3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/o/openssl/openssl_3.0.2-0ubuntu1.6_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/pci.ids/pci.ids_0.0~2022.01.22-1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/pciutils/libpci3_3.7.0-6_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/pmdk/libpmem1_1.11.1-3build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/pmdk/libpmemobj1_1.11.1-3build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/popt/libpopt0_1.18-3build1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/prettytable/python3-prettytable_2.5.0-2_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/psmisc/psmisc_23.4-2build3_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/python-certifi/python3-certifi_2020.6.20-1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/python-idna/python3-idna_3.3-1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/python-urllib3/python3-urllib3_1.26.5-1~exp1_all.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.com/ubuntu/pool/main/p/pyyaml/python3-yaml_5.4.1-1ubuntu1_amd64.deb': Read-only file system
rm: cannot remove '/tmp/kubekey/iso/archive.ubuntu.co

i must umount /tmp/kubekey/iso by hand

problems 2


:46:05 CST success: [harbor]
20:46:05 CST [RepositoryModule] New repository client
20:46:05 CST success: [harbor]
20:46:05 CST success: [node1]
20:46:05 CST success: [node2]
20:46:05 CST success: [node3]
20:46:05 CST [RepositoryModule] Backup original repository
20:46:05 CST message: [node1]
backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
20:46:05 CST message: [node2]
backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
20:46:05 CST message: [node3]
backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
20:46:05 CST message: [harbor]
backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
20:46:05 CST failed: [node1]
20:46:05 CST failed: [node2]
20:46:05 CST failed: [node3]
20:46:05 CST failed: [harbor]
20:46:05 CST rollback: [harbor]
20:46:05 CST rollback: [node3]
20:46:05 CST rollback: [node2]
20:46:05 CST rollback: [node1]
error: Pipeline[CreateClusterPipeline] execute failed: Module[RepositoryModule] exec failed: 
failed: [node1] [BackupOriginalRepository] exec failed after 1 retires: backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
failed: [node2] [BackupOriginalRepository] exec failed after 1 retires: backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
failed: [node3] [BackupOriginalRepository] exec failed after 1 retires: backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1
failed: [harbor] [BackupOriginalRepository] exec failed after 1 retires: backup repository failed: Failed to exec command: sudo -E /bin/bash -c "mv /etc/apt/sources.list /etc/apt/sources.list.kubekey.bak" 
mv: cannot stat '/etc/apt/sources.list': No such file or directory: Process exited with status 1

willzhang avatar Sep 08 '22 12:09 willzhang

Obviously, you manually terminated the kk, resulting in that's kk rollback operation was not executed.

24sama avatar Sep 08 '22 13:09 24sama

rollback maybe should Idempotent like ansible though i manually terminated kk.

willzhang avatar Sep 13 '22 02:09 willzhang

That is a good enhancement for kk command line.

/kind feature-request

24sama avatar Sep 14 '22 02:09 24sama

I also thought it was interrupted during execution. How did you solve it later?

1764115099 avatar Oct 11 '23 09:10 1764115099