CI failure: spider pod restarted

[weizhoublue]

Set title to be in the format CI: spider pod restarted

Copy-paste the output the test failure.

the error happened after reboot node test case , please look into whether there is any bug owing to spider pod restart (1)BTW: reboot node test case must wait for all pod running (2) please look into how long spiderpool-agent and spiderpool-controller need change to be ready

-------- kubectl get pod -A -o wide
NAMESPACE          NAME                                                         READY   STATUS              RESTARTS      AGE     IP              NODE                                 NOMINATED NODE   READINESS GATES
default            test-pod-6c5cdc6fb6-qd854                                    1/1     Running             0             7m8s    172.18.40.173   spiderpool0817135416-control-plane   <none>           <none>
kube-system        coredns-78fcd69978-gh6jh                                     0/1     Unknown             0             83s     <none>          spiderpool0817135416-worker          <none>           <none>
kube-system        coredns-78fcd69978-z9m5h                                     0/1     Unknown             0             83s     <none>          spiderpool0817135416-worker          <none>           <none>
kube-system        etcd-spiderpool0817135416-control-plane                      1/1     Running             0             2m10s   172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        kube-apiserver-spiderpool0817135416-control-plane            1/1     Running             0             60s     172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        kube-controller-manager-spiderpool0817135416-control-plane   1/1     Running             0             9m20s   172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        kube-multus-ds-75w5n                                         1/1     Running             1 (31s ago)   7m19s   172.18.0.3      spiderpool0817135416-worker          <none>           <none>
kube-system        kube-multus-ds-jhrhc                                         1/1     Running             0             7m19s   172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        kube-proxy-ls7q4                                             1/1     Running             1 (31s ago)   8m54s   172.18.0.3      spiderpool0817135416-worker          <none>           <none>
kube-system        kube-proxy-wl95z                                             1/1     Running             0             9m7s    172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        kube-scheduler-spiderpool0817135416-control-plane            1/1     Running             0             9m20s   172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        spiderpool-agent-9x7pf                                       1/1     Running             0             109s    172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
kube-system        spiderpool-agent-njr7s                                       0/1     Running             1 (31s ago)   110s    172.18.0.3      spiderpool0817135416-worker          <none>           <none>
kube-system        spiderpool-controller-74449fb8-lk85p                         1/1     Running             0             70s     172.18.0.2      spiderpool0817135416-control-plane   <none>           <none>
ns2014-222661453   pod2014-230069359                                            0/1     Terminating         0             70s     <none>          spiderpool0817135416-worker          <none>           <none>
ns2023-327784506   pod2023-337690446                                            0/1     Terminating         0             60s     <none>          spiderpool0817135416-worker          <none>           <none>
ns2041-457927411   pod2041-467356546-7ff6d7f8-wrmlt                             0/1     Terminating         0             45s     <none>          spiderpool0817135416-worker          <none>           <none>
ns2041-457927411   pod2041-467356546-7ff6d7f8-wtssj                             0/1     Terminating         0             45s     <none>          spiderpool0817135416-worker          <none>           <none>
ns2054-759821059   pod2054-780794559                                            0/1     ContainerCreating   0             32s     <none>          spiderpool0817135416-worker          <none>           <none>
ns2054-900740677   pod2054-970132472                                            0/1     ContainerCreating   0             32s     <none>          spiderpool0817135416-worker          <none>           <none>

pod exited with 255

---------kubectl describe pod spiderpool-agent-njr7s -n kube-system 
Name:                 spiderpool-agent-njr7s
Namespace:            kube-system
Priority:             2000001000
Priority Class Name:  system-node-critical
Node:                 spiderpool0817135416-worker/172.18.0.3
Start Time:           Wed, 17 Aug 2022 14:19:36 +0000
Labels:               app.kubernetes.io/component=spiderpoolagent
                      app.kubernetes.io/instance=spiderpool
                      app.kubernetes.io/name=spiderpool
                      controller-revision-hash=5f6b699dbd
                      pod-template-generation=2
Annotations:          <none>
Status:               Running
IP:                   172.18.0.3
IPs:
  IP:           172.18.0.3
Controlled By:  DaemonSet/spiderpool-agent
Containers:
  spiderpool-agent:
    Container ID:  containerd://051dece877f09ec9ef4d613ba08fde47008bfa214c7bcd271beb666b1c526ed8
    Image:         spiderpool-agent-race:56c8b4b5eedee72eb91b5d3a447a4f9c7ea68ea8
    Image ID:      docker.io/library/import-2022-08-17@sha256:192689f05dd96a2bafc607f0cc6c364505c6b7641ad8d3b8c6bce8cf688b63b2
    Port:          <none>
    Host Port:     <none>
    Command:
      spiderpool-agent
    Args:
      daemon
      --config-path=/tmp/spiderpool/config-map/conf.yml
    State:          Running
      Started:      Wed, 17 Aug 2022 14:20:58 +0000
    Last State:     Terminated
      Reason:       Unknown
      Exit Code:    255
      Started:      Wed, 17 Aug 2022 14:19:37 +0000
      Finished:     Wed, 17 Aug 2022 14:20:55 +0000

Upload the zip file generated from that test failure. e2edebugLog 2.txt

Copy-paste the link of the CI build where that test failure has happen. https://github.com/spidernet-io/spiderpool/runs/7880191548?check_suite_focus=true

Include any output from logs that you think may be relevant (to ease GitHub searches).