kubeadm installed cluster: control-plane pods does not show up in pixie

[knfoo]

Describe the bug The Pixie UI does not contain the control-plane pods from a kubeadm installed k8s cluster

To Reproduce Steps to reproduce the behavior:

1. Install a Kubernetes cluster with kubeadm
2. Install px to the cluster
3. Navigate to https://work.withpixie.ai/live/clusters//namespaces/kube-system?start_time=-5m
4. See the list of pods does not contain:

kubectl -n kube-system get pods -ltier=control-plane
etcd
kube-apiserver
kube-controller-manager
kube-scheduler

Expected behavior That all pods in the kubernetes cluster shows up in the Pixie UI

Logs pixie_logs_20220119195505.zip

App information (please complete the following information):

• Pixie version: v0.10.2
• K8s cluster version 1.21.5
• Node Kernel version 5.15.14
• Browser version FF

Additional context https://pixie-community.slack.com/archives/CQ63KEVFY/p1641220096155500

[vincent-pli]

Same question, is that by design?

[vincent-pli]

Also there is nothing in the output of table: http_events related to api-server.

I try to install a new operator, expect to see the connection between it and api-server(by informer from operator to api-server), but I only can see the /health or /readiness probe request to the operator from kubelet:

and the source IP is my private node IP and the port is random seems.

but I can see the connection in the table: conn_stats between operators and api-server by the informer.

my env is also setup by kubeadm, is that the reason?

[vincent-pli]

Could anyone explain the question, I install another pixie with minikube, same issue @vihangm @oazizi000 @aimichelle

[aimichelle]

Hi @knfoo , we did some digging to see what's going on and found a fundamental difference between how normal K8s pods and control-plane pods process stats are stored. For most cases, the cgroup path for k8s pods is something like /sys/fs/cgroup/cpu,cpuacct/kubepods/pod<Pod UID>. However, this is different for control-plane pods, where the path is instead /sys/fs/cgroup/cpu,cpuacct/kubepods/pod<config hash>, where the config hash matches a K8s annotation on the control-plane pod.

We'll likely need to update Vizier so that it sends that config hash to the PEMs, but will also need to do some more investigation about whether that is actually the right path moving forward (relying on an pod annotation seems risky).

@vincent-pli , I just want to confirm that I understand your issue correctly. Is this for the kube-apiserver?

[vincent-pli]

Thanks @aimichelle

The records of snapshot in the previously comment are http flow related to my new controller I expect to see some flow between it and kube-apiserver, but only have some one between it and kubelet(readiness probe)

That's great you can address the issue, thanks again.

[vincent-pli]

@aimichelle I did some checking, seems the problem is related to static pod, I prefer to open a new issue to trace it.