k8s-gitops
k8s-gitops copied to clipboard
billimek
high CPU usage for k3s-server
Any chance you've run into this issue?
https://github.com/rancher/k3s/issues/294
I've noticed that my nzbget client doesn't get any more than 5-8MB/s. Node running the pod has a 13/13/13 on load with 4 procs. Turns out that node is also the master and k3s-server process is destroying the local cpu:
I think I may try an A/B test with another rke cluster to see if it's any better.
I just looked and see the k3s-agent process running around 30-45% CPU usage on the nodes I have it installed. Not sure if this is 'normal' or not. I'm subscribed to the issue you linked and curious where the problem may be.
For some comparison, here is a 30-day view of CPU usage for the 7 k3s nodes I run:
As an aside, the CPU usage seems a bit higher after moving from raspbian to ubuntu 19.10 (arm64) on the pi4 nodes
Unfortunately the other issue was closed. I suppose we would need some test setup using vagrant to prove that k3s consumes more cpu than a similar setup using plain kubeadm or microk8s.
Also wondering if the high load may be tied to docker/container networking. I've now proven out that running nzbget on a VM with an NFS mount for storage gets me close to 100MB/s on gigabit internet.
Running the exact same configuration on the same VM with the same host with the same NFS share in a docker container maxes out at 8MB/s. This test node isn't apart of my k3s cluster, so there's no overlay networking going on. Also seeing the high system load experienced within the k3s cluster.
and, just for grins, I tried it with Macvlan, same problem. I wonder if there's something with the ubuntu cloud image kernel?
@carpenike are you running all raspberry pis with ubuntu?
I started looking closer and it seems, for me, after switching from raspbian to ubuntu 19.10 (arm64), they run a lot higher CPU:
raspbian, 48-hour view:
ubuntu 19.10 (arm64) 48-hour view:
Another point of comparison is a 2-day snapshot of CPU & load compared side-by-side with a node still on raspbian (pi4-b) and a node on ubuntu 19.10 (arm64) (pi4-a):
It's too bad they closed the issue on the k3s repo. I'm seeing a pretty steady 30% CPU consumption of the k3s agent on a pi4 node which is cordoned in the cluster, so no workloads are really running there:
@billimek would you be able to provide some vagrant setup which would test k3s, kubeadm and microk8s in order to prove that something is broken in k3s? The developers mentioned that without some test setup which shows big differences they do not see any problem....
This is also interesting, going to dig in a bit more but it appears that all the nodes tunnel traffic through the master?
https://reddit.com/r/kubernetes/comments/bj2ebf/_/em55ix4/?context=1
Do you have many workloads which use an exec based liveness/readiness probes? I came across this issue and wonder if there could be some relationship?
I can look into it. Don't we see the same problems with a cordon'ed host though?
The same on Ubuntu 18, even after clean install - about 12% CPU usage in top goes to k3s-server process.
