pixie-io
Pixie Misses Events from Short Lived Processes/Pods
Describe the bug In many cases, Pixie fails to report HTTP events for short-lived processes, including for short-lived Kubernetes Pods and for short-lived child-processes within longer-living Pods. This is because Pixie's data collector Stirling fails to collect the necessary process metadata. This metadata is used to reconcile communication traces with their source/destination pod(s).
Currently, running processes have their process id (pid) communicated to Stirling by each data capture ebpf probe. In order to properly associate this pid with a kubernetes pod, a lookup procedure in the Stirling's userspace periodically probes Linux's sysfs in order to collect all pids in each cgroup, along with that cgroup's metadata. The procedure to associate a pid with a kubernetes pod does a lookup using the pid to find the cgroup containing the process. The cgroup path name is related to the pod unique id (pod uid) and they are able to match them using regex (code here). However pids of short-lived processes will leave the cgroup once the process terminates. Thus if the process terminates before a lookup occurs, then the data for that pid cannot be exported from stirling.
To Reproduce Steps to reproduce the behavior:
- Deploy pixie
- Start a container and ssh into it
- Run
curl example.com - Check Pixie Dashboard and you'll see that the behavior was not captured in the http_data script.
Expected behavior The Behavior of Short Running processes should be captured. There are several ways to achieve this but this is the desired high level behavior.
App information (please complete the following information): This is an issue discussed with the Pixie team, and impacts pixie in all its current versions.
Additional Information This is a known issue to the Pixie Team, and I am working with them to fix it. Here is a link to the evolving design document.
