tetragon icon indicating copy to clipboard operation
tetragon copied to clipboard

Q: uprobes in cluster

Open n-bes opened this issue 8 months ago • 4 comments

Hi.

I'm a little confused working with uprobes tracing policy.

python scenarion

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: "python3-uprobe"
spec:
  uprobes:
  - path: "/usr/bin/python3"
    symbols:
      - "PyObject_ASCII"
      - "PyObject_AsCharBuffer"

In my case, it is crashed because /usr/bin/python3 not exists. Also I mentioned that it is compiles some bpf program against binary while applying the policy.

  • Could describe, how it works?
  • What if multiply binaries exists? (Pythoninv 3.11 / 3.12)
  • Does this require a running application? Compiles against all binaries matched by name?

bash scenarion

I'm back to example from docs: https://tetragon.io/docs/concepts/tracing-policy/hooks/#uprobes. I added all available functions (~1k). It works but required a lot of time (~30 min) for preparing. Could it be faster?

n-bes avatar Mar 31 '25 03:03 n-bes

to create uprobe we need access to the binary (inode), there's a way to specify path out of the container to the host, forgot the details (@kkourt would you have a link?)

I'm back to example from docs: https://tetragon.io/docs/concepts/tracing-policy/hooks/#uprobes.

you can generate uprobe policy via: # tetra tracingpolicy generate uprobes --binary /bin/bash

will add it to the docs

I added all available functions (~1k). It works but required a lot of time (~30 min) for preparing. Could it be faster?

v6.6 kernels have support for uprobe multi link and speed up above a lot

olsajiri avatar Mar 31 '25 07:03 olsajiri

will add it to the docs

@olsajiri thx for fast reply. Please describe cases:

  • k8s
  • different binaries with same name

n-bes avatar Mar 31 '25 07:03 n-bes

To create uprobe we need access to the binary (inode), there's a way to specify path out of the container to the host, forgot the details (@kkourt would you have a link?)

I'm not aware of an easy way to do this in a cluster. If you are on the host on a node, you can run something like:

#  crictl inspect d545013c4b6af | jq .info.runtimeSpec.root

To get the root filesystem of a container, and then add the policy path there.

kkourt avatar Mar 31 '25 07:03 kkourt

* different binaries with same name

such binaries will have different inode and path, so you need to setup for each such binary

olsajiri avatar Mar 31 '25 07:03 olsajiri

I think the questions here were replied, I'll close it for now. We could open new issues for ideas on improvements for uprobes on k8s. Feel free to reopen/ask questions here if still needed.

mtardy avatar Jul 15 '25 16:07 mtardy

I think the questions here were replied, I'll close it for now

@kkourt @olsajiri @mtardy thank you

I will wait for the solution of https://github.com/cilium/tetragon/issues/3571

n-bes avatar Jul 15 '25 17:07 n-bes