rancher [BUG] k3s node driver cluster stuck in updating state after k3s version upgrade

[BUG] k3s node driver cluster stuck in updating state after k3s version upgrade

Open sowmyav27 opened this issue 2 years ago • 1 comments

Rancher Server Setup

Rancher version: 2.6.7
Installation option (Docker install/Helm Chart): docker install

Information about the Cluster

Kubernetes version: 1.23.8+k3s2 to v1.24.2+k3s2
Cluster Type (Local/Downstream): downstream - Azure node driver - 3 etcd, 1 cp and 9 worker nodes

User Information

What is the role of the user logged in? (Admin/Cluster Owner/Cluster Member/Project Owner/Project Member/Custom) admin

Describe the bug [BUG] k3s node driver cluster stuck in updating state after k3s version upgrade

To Reproduce

Deploy a node driver cluster on 1.23 k3s version
i scaled down a worker node ( was part of another test case) --> this seems to cause this issue, without this step, the k3s upgrade was good.
Upgrade k3s version to 1.24 k3s version.
The cluster went into updating state and the cluster is stuck in that state.
Two of the three ~~worker~~ etcd nodes are NOT active
Error on the etcd nodes Node condition MemoryPressure is Unknown. Node condition DiskPressure is Unknown. Node condition PIDPressure is Unknown. Node condition Ready is Unknown. in the Rancher UI.
Provisioning logs:



10:02:02 am | [INFO ] waiting for viable init node
-- | --
10:02:32 am | [INFO ] configuring bootstrap node(s) sowmya-new-pool1-6884b9bb89-qptsc: waiting for agent to check in and apply initial plan
10:03:58 am | [INFO ] configuring etcd node(s) sowmya-new-pool1-6884b9bb89-l2dxv,sowmya-new-pool1-6884b9bb89-sd78n
10:04:16 am | [INFO ] configuring etcd node(s) sowmya-new-pool1-6884b9bb89-l2dxv: waiting for agent to check in and apply initial plan
10:04:32 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for agent to check in and apply initial plan
10:04:46 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for probes: kube-controller-manager, kube-scheduler, kubelet
10:04:48 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for probes: kube-scheduler, kubelet
10:04:52 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for probes: kube-scheduler
10:04:56 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for cluster agent to connect
10:06:02 am | [INFO ] waiting for control plane to be available
10:06:22 am | [ERROR] Operation cannot be fulfilled on machines.cluster.x-k8s.io "sowmya-new-pool3-5bd49457c-2z9nw": the object has been modified; please apply your changes to the latest version and try again
10:06:22 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-9wx4v and 7 more
10:06:34 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-dmvhc and 6 more
10:06:38 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-plzxr,sowmya-new-pool3-5bd49457c-qf7ps and 3 more
10:06:40 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-rjnwb,sowmya-new-pool3-5bd49457c-s5mbq,sowmya-new-pool3-5bd49457c-ssbtq
10:06:42 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-9wx4v and 6 more
10:06:52 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-l5hf5 and 5 more
10:06:54 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-plzxr and 4 more
10:06:56 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-rjnwb and 2 more
10:06:58 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-rjnwb,sowmya-new-pool3-5bd49457c-s5mbq and 1 more
10:07:00 am | [INFO ] non-ready worker machine(s) sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-rjnwb,sowmya-new-pool3-5bd49457c-ssbtq
10:07:02 am | [INFO ] provisioning done
10:13:28 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for plan to be applied
10:13:34 am | [INFO ] provisioning done
10:14:20 am | [INFO ] configuring bootstrap node(s) sowmya-new-pool1-6884b9bb89-qptsc: waiting for plan to be applied
10:14:36 am | [INFO ] configuring bootstrap node(s) sowmya-new-pool1-6884b9bb89-qptsc: waiting for kubelet to update
10:14:38 am | [INFO ] configuring etcd node(s) sowmya-new-pool1-6884b9bb89-l2dxv,sowmya-new-pool1-6884b9bb89-sd78n
10:14:50 am | [INFO ] configuring etcd node(s) sowmya-new-pool1-6884b9bb89-sd78n: waiting for plan to be applied
10:15:02 am | [INFO ] configuring etcd node(s) sowmya-new-pool1-6884b9bb89-sd78n: waiting for kubelet to update
10:15:04 am | [INFO ] sowmya-new-pool1-6884b9bb89-l2dxv,sowmya-new-pool1-6884b9bb89-sd78n
10:15:06 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for plan to be applied
10:15:20 am | [INFO ] configuring control plane node(s) sowmya-new-pool2-6cf484c6c7-6vpmq: waiting for kubelet to update
10:15:22 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-dmvhc and 6 more
10:15:32 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-dmvhc,sowmya-new-pool3-5bd49457c-l5hf5 and 5 more
10:15:42 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-dmvhc,sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-plzxr and 4 more
10:15:52 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-l5hf5: waiting for plan to be applied
10:15:54 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-plzxr,sowmya-new-pool3-5bd49457c-qf7ps and 3 more
10:15:58 am | [ERROR] Operation cannot be fulfilled on machines.cluster.x-k8s.io "sowmya-new-pool3-5bd49457c-l5hf5": the object has been modified; please apply your changes to the latest version and try again
10:16:00 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-l5hf5,sowmya-new-pool3-5bd49457c-plzxr,sowmya-new-pool3-5bd49457c-qf7ps and 3 more
10:16:08 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-plzxr,sowmya-new-pool3-5bd49457c-qf7ps,sowmya-new-pool3-5bd49457c-rjnwb and 2 more
10:16:26 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-qf7ps,sowmya-new-pool3-5bd49457c-rjnwb,sowmya-new-pool3-5bd49457c-s5mbq and 1 more
10:16:36 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-rjnwb,sowmya-new-pool3-5bd49457c-s5mbq,sowmya-new-pool3-5bd49457c-ssbtq
10:16:46 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-s5mbq,sowmya-new-pool3-5bd49457c-ssbtq
10:16:56 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-ssbtq: waiting for plan to be applied
10:17:00 am | [INFO ] configuring worker node(s) sowmya-new-pool3-5bd49457c-ssbtq: waiting for probes: kubelet
10:17:10 am | [INFO ] sowmya-new-pool3-5bd49457c-2z9nw,sowmya-new-pool3-5bd49457c-7mnd2,sowmya-new-pool3-5bd49457c-dmvhc and 6 more
10:17:12 am | [INFO ] failing etcd machine(s) sowmya-new-pool1-6884b9bb89-l2dxv,sowmya-new-pool1-6884b9bb89-sd78n

journalctl -u rancher-system-agent logs

-- Logs begin at Sat 2022-08-20 17:02:28 UTC, end at Sat 2022-08-20 17:19:42 UTC. --
Aug 20 17:03:58 sowmya-new-pool1-95bbf824-99f4n systemd[1]: Started Rancher System Agent.
Aug 20 17:03:58 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:58Z" level=info msg="Rancher System Agent version v0.2.10 (6a89e30) is starting"
Aug 20 17:03:58 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:58Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Aug 20 17:03:58 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:58Z" level=info msg="Starting remote watch of plans"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="Starting /v1, Kind=Secret controller"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="Detected first start, force-applying one-time instruction set"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.23.8-k3s2 to directory /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Aug 20 17:03:59 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:03:59Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.23.8-k3s2"
Aug 20 17:04:00 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:00Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/installer.sh"
Aug 20 17:04:00 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:00Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/run.sh"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + cp -f /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/k3s /usr/local/bin/k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + chmod 755 /usr/local/bin/k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + chown root:root /usr/local/bin/k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + mkdir -p /var/lib/rancher/k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -f /var/lib/rancher/k3s/restart_stamp ]"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -n 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [  != 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + FORCE_RESTART=true"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + env INSTALL_K3S_FORCE_RESTART=true INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping k3s download and verify"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping installation of SELinux RPM"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating /usr/local/bin/kubectl symlink to k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating /usr/local/bin/crictl symlink to k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating /usr/local/bin/ctr symlink to k3s"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  systemd: Enabling k3s unit"
Aug 20 17:04:01 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:01Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service."
Aug 20 17:04:02 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:02Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  systemd: Starting k3s"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -n 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + echo 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220820-170359/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: Name Location Size Created"
Aug 20 17:04:31 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[1405]: time="2022-08-20T17:04:31Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Aug 20 17:07:09 sowmya-new-pool1-95bbf824-99f4n systemd[1]: Stopping Rancher System Agent...
Aug 20 17:07:09 sowmya-new-pool1-95bbf824-99f4n systemd[1]: rancher-system-agent.service: Succeeded.
Aug 20 17:07:09 sowmya-new-pool1-95bbf824-99f4n systemd[1]: Stopped Rancher System Agent.
Aug 20 17:07:11 sowmya-new-pool1-95bbf824-99f4n systemd[1]: Started Rancher System Agent.
Aug 20 17:07:11 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:11Z" level=info msg="Rancher System Agent version v0.2.10 (6a89e30) is starting"
Aug 20 17:07:11 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:11Z" level=info msg="Using directory /var/lib/rancher/agent/work for work"
Aug 20 17:07:11 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:11Z" level=info msg="Starting remote watch of plans"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Starting /v1, Kind=Secret controller"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Detected first start, force-applying one-time instruction set"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.23.8-k3s2 to directory /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.23.8-k3s2"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/installer.sh"
Aug 20 17:07:12 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:12Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/k3s"
Aug 20 17:07:14 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:14Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/run.sh"
Aug 20 17:07:14 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:14Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Aug 20 17:07:14 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:14Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + cp -f /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0/k3s /usr/local/bin/k3s"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + chmod 755 /usr/local/bin/k3s"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + chown root:root /usr/local/bin/k3s"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + mkdir -p /var/lib/rancher/k3s"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -f /var/lib/rancher/k3s/restart_stamp ]"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + cat /var/lib/rancher/k3s/restart_stamp"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + PRIOR_RESTART_STAMP=86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -n 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a != 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + FORCE_RESTART=false"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + env INSTALL_K3S_FORCE_RESTART=false INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping k3s download and verify"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping installation of SELinux RPM"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh"
Aug 20 17:07:16 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:16Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh"
Aug 20 17:07:17 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:17Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env"
Aug 20 17:07:17 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:17Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service"
Aug 20 17:07:17 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:17Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  systemd: Enabling k3s unit"
Aug 20 17:07:17 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:17Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service."
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + [ -n 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a ]"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stderr]: + echo 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: [INFO]  No change detected so skipping service start"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220820-170712/426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[426a416175c7130ab7414e0cbd9484066bbeedb612cb7da135978581d36b8f52_0:stdout]: Name Location Size Created"
Aug 20 17:07:18 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:07:18Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"
Aug 20 17:14:37 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:37Z" level=info msg="[Applyinator] Applying one-time instructions for plan with checksum 7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95"
Aug 20 17:14:37 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:37Z" level=info msg="[Applyinator] Extracting image rancher/system-agent-installer-k3s:v1.24.2-k3s2 to directory /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0"
Aug 20 17:14:37 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:37Z" level=info msg="Using private registry config file at /etc/rancher/agent/registries.yaml"
Aug 20 17:14:37 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:37Z" level=info msg="Pulling image index.docker.io/rancher/system-agent-installer-k3s:v1.24.2-k3s2"
Aug 20 17:14:38 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:38Z" level=info msg="Extracting file installer.sh to /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0/installer.sh"
Aug 20 17:14:38 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:38Z" level=info msg="Extracting file k3s to /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0/k3s"
Aug 20 17:14:39 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:39Z" level=info msg="Extracting file run.sh to /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0/run.sh"
Aug 20 17:14:39 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:39Z" level=info msg="[Applyinator] Running command: sh [-c run.sh]"
Aug 20 17:14:39 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:39Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + cp -f /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0/k3s /usr/local/bin/k3s"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + chmod 755 /usr/local/bin/k3s"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + chown root:root /usr/local/bin/k3s"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + mkdir -p /var/lib/rancher/k3s"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + RESTART_STAMP_FILE=/var/lib/rancher/k3s/restart_stamp"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + [ -f /var/lib/rancher/k3s/restart_stamp ]"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + cat /var/lib/rancher/k3s/restart_stamp"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + PRIOR_RESTART_STAMP=86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + [ -n b7fb45121f505de5c096bcba6f9bdb7cf4e6bfdd62351745f6f49ac3f725300b ]"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + [ 86670529b91fce5b180c0e4044333e36dde7ec81bf8ed59b36aff8aeb53a7f6a != b7fb45121f505de5c096bcba6f9bdb7cf4e6bfdd62351745f6f49ac3f725300b ]"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + FORCE_RESTART=true"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + env INSTALL_K3S_FORCE_RESTART=true INSTALL_K3S_SKIP_DOWNLOAD=true INSTALL_K3S_SKIP_SELINUX_RPM=true INSTALL_K3S_SELINUX_WARN=true installer.sh"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Skipping k3s download and verify"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Skipping installation of SELinux RPM"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Skipping /usr/local/bin/kubectl symlink to k3s, already exists"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Skipping /usr/local/bin/crictl symlink to k3s, already exists"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Skipping /usr/local/bin/ctr symlink to k3s, already exists"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Creating killall script /usr/local/bin/k3s-killall.sh"
Aug 20 17:14:40 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:40Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  Creating uninstall script /usr/local/bin/k3s-uninstall.sh"
Aug 20 17:14:41 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:41Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  env: Creating environment file /etc/systemd/system/k3s.service.env"
Aug 20 17:14:41 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:41Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  systemd: Creating service file /etc/systemd/system/k3s.service"
Aug 20 17:14:41 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:41Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  systemd: Enabling k3s unit"
Aug 20 17:14:41 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:41Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: Created symlink /etc/systemd/system/multi-user.target.wants/k3s.service → /etc/systemd/system/k3s.service."
Aug 20 17:14:42 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:42Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: [INFO]  systemd: Starting k3s"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + [ -n b7fb45121f505de5c096bcba6f9bdb7cf4e6bfdd62351745f6f49ac3f725300b ]"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stderr]: + echo b7fb45121f505de5c096bcba6f9bdb7cf4e6bfdd62351745f6f49ac3f725300b"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[Applyinator] Command sh [-c run.sh] finished with err: <nil> and exit code: 0"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[Applyinator] No image provided, creating empty working directory /var/lib/rancher/agent/work/20220820-171437/7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[Applyinator] Running command: sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null]"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[7e27b7c01eac2bca44d9abc73df5145c2998e20d244a1d79cec4af005f1f7c95_0:stdout]: Name Location Size Created"
Aug 20 17:14:45 sowmya-new-pool1-95bbf824-99f4n rancher-system-agent[5597]: time="2022-08-20T17:14:45Z" level=info msg="[Applyinator] Command sh [-c k3s etcd-snapshot list --etcd-s3=false 2>/dev/null] finished with err: <nil> and exit code: 0"

Note:

This is NOT seen on an RKE2 cluster upgrade from 1.23 to 1.24
and with the steps mentioned above ^^ I was able to reproduce the issue 2 out of 2 times.

Aug 20 '22 17:08 sowmyav27

Did you ever figure this out? Having exactly the same issue on Debian 11 install when provisioning new 1.24.4+rke2r1 from rancher.

Sep 13 '22 03:09 martinkeat

jameson-mcghee said: When testing v2.6.7 and on v2.7-head (Commit ID: 314e690) using the following Steps to Reproduce, I did not observe the issue. All clusters that I attempted to reproduce this issue on were able to come back to Active after the k8s upgrade. cc @sowmyav27

Steps to Reproduce:

Deploy an RKE2 Node Driver cluster on k8s v1.23.8+k3s2 with 3 etcd, 2 control plane, and 9 worker nodes (in separate pools)
Once the cluster is Active, Scale Down one of the worker nodes
Once the cluster is Active again, using Edit Config, change the k8s version to v1.24.2+k3s2 -The cluster will never come back up Active, but instead stays in an Updating state with one of the etcd nodes remaining inactive (Purported Issue, not observed during my testing)

v2.6.7 (post-k8s-upgrade):

v2.7-head (post-k8s-upgrade):

Oct 04 '22 20:10 jameson-mcghee

Closing this issue(I wasn't able to reproduce it on 2.7-head either), will log a new one if we hit this issue again.

Oct 05 '22 00:10 sowmyav27

rancher rancher copied to clipboard

[BUG] k3s node driver cluster stuck in updating state after k3s version upgrade

rancher
rancher copied to clipboard