Apply Policy Only to Specific Containers in a Pod

[BonySmoke]

Is there an existing issue for this?

• [X] I have searched the existing issues

Is your feature request related to a problem?

Currently, we can apply a policy to a specific set of pods using the podSelector block. We want to apply a policy only to certain container(s) inside a pod/pods.

In our use case, we have a main and a sidecar container inside a pod. We want to block write requests to files under a particular directory in the main container - let's say /home/user. The problem is that the sidecar container may write to files in the same path as the main container and we want to allow it.

We tried to understand if we can differentiate containers based on binaries that perform write requests or PIDs. Unfortunately, either container can use the same binary and we couldn't understand how to differentiate containers based on the PID.

Please let us know if we are missing something.

Describe the feature you would like

It would be great to be able to apply a policy only to specific containers inside a pod/pods. As I can see, the information about the container is already present in JSON events.

Describe your proposed solution

First of all, I want to say that I'm not familiar with the source code well enough to propose exact solutions, but I was thinking about the following 2 approaches:

1. Add a separate containerSelector block. It can be used independently to apply the policy to all containers in the cluster with specified names, as well as with podSelector limiting containers to a specific pod/pods.

spec:
  containerSelector:
    operator: In
    values:
    - main
    - secondary
  # If podSelector is present, the policy applies only to containers specified above in this pod
  podSelector:
    matchLabels:
      app: "my-awesome-app"

2. If the approach above is not optimal or not feasible, maybe it's possible to add a matchContainers selector (similar to matchBinaries, etc.)?

- matchContainers:
  - operator: "In"
    values:
    - "main"
    - "secondary"

Please let me know if none of the proposed solutions make sense and thank you in advance! :smiley:

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[mtardy]

Thanks for the detailed request, I personally think it makes sense to have the container granularity at first glance. Maybe the first proposition would make sense as the mechanism to filter things by namespace/labels could be adapted to containers in pods.

I'll let @kkourt take a look at this as he implemented the namespace/label filtering.

[kkourt]

Thanks! I think that's a reasonable feature to have.

Not sure what the proper interface for the policy would be. Would need to think a bit more about it.

In terms of implementation, it's certainly something doable. We already track containers for pod-label filters so we would just need to additional filtering based on the container name.

See:

https://github.com/cilium/tetragon/blob/908f4db1545b37f8d1ad9781a7b407a23830ea04/pkg/policyfilter/state.go#L765

And

https://github.com/cilium/tetragon/blob/908f4db1545b37f8d1ad9781a7b407a23830ea04/pkg/policyfilter/state.go#L565

For some examples.

[sadath-12]

/assign

[BonySmoke]

Hello everyone! We want to try to implement this feature request and we would like to discuss a couple of points first :slightly_smiling_face:

@sadath-12 I see you assigned this feature to yourself; I want to ensure we will not interrupt your work. Please let me know if you are already working on it and plan to finish it; otherwise, hopefully, you won't mind us taking it:)

Next, I would like to understand an optimal way of implementing this feature. @kkourt wasn't sure about the proposed interface from the issue description; therefore, I think it's worth agreeing on some solution before implementing it.

However, if we stick to the proposed solution with containerSelector, I was thinking about the following steps:

• In policy, add containerSelector, the value will be passed when updating policy filter.
• I believe we will need to implement/reuse some functions to do name matching just like with labels.
• Actual Filtering: Currently, when creating a policy, we check if the pod matches the policy filter, then, we go through every container and this is where we could add one more containerMatches check and if it's true, we add the container to the list; otherwise, we skip it. If I understand it correctly, the policy is applied by using the cgroup of a container and this approach should work. However, please let me know if I misunderstood it.
• Also, we will need to handle policy refreshes. addPodContainers refreshes the policy map and adds new containers. We would need to add the same container filtering logic before adding new container IDs here.

What do you think, does this plan look feasible? Let me know if we are missing something obvious or if you have any other ideas:)

[mtardy]

@sadath-12 I see you assigned this feature to yourself; I want to ensure we will not interrupt your work. Please let me know if you are already working on it and plan to finish it; otherwise, hopefully, you won't mind us taking it:)

It's okay you should be able to work on this as the assignment was two months ago and I don't think there's a draft PR out there. We try to avoid cookie licking.

Thanks for your detailed message, Kornilios can indeed help you more than me (I would need to dive into this properly), but since you did significant research, feel free to open a draft PR with your ideas, it might be easier to discuss if you have a POC or an early implementation.

[kkourt]

Hello everyone! We want to try to implement this feature request and we would like to discuss a couple of points first 🙂

[...] What do you think, does this plan look feasible? Let me know if we are missing something obvious or if you have any other ideas:)

Thanks! Everything makes sense to me and plan seems great :)

[kkourt]

One thing that I would maybe like to discuss is the syntax for the container selector. The main use-case right now is to select based on the name. It would be nice if whatever syntax we come up with does not preclude selecting based on other properties of the container object.

How about adding a type in the selector and make it a list?

  containerSelector:
   - type: nameSelector # just an example :)
     operator: In
     values:
     - main
     - secondary

[BonySmoke]

One thing that I would maybe like to discuss is the syntax for the container selector. The main use-case right now is to select based on the name. It would be nice if whatever syntax we come up with does not preclude selecting based on other properties of the container object.

How about adding a type in the selector and make it a list?

  containerSelector:
   - type: nameSelector # just an example :)
     operator: In
     values:
     - main
     - secondary

Thanks a lot for this valuable comment! I agree, this sounds great and it adds a lot of flexibility! I was thinking about a very similar thing, but instead of type use key, and it would be the field in the container spec that we filter by, e.g. name, image, etc. Also, it would be compatible with the matchExpressions schema so it would be easier for people to read it. Therefore, maybe it would look like this:

containerSelector:
  matchExpressions:
    - key: name
      operator: In
      values:
        - main
        - secondary

The benefit I see in this structure is that it will be compatible with the code for label filtering:)

Just wanted to say that my Go developer skills are suboptimal (for now:) ); therefore, I'm learning things along the way. At the moment, I'm trying to follow Mahé's suggestion and create an early implementation.

[BonySmoke]

Hello everyone!:) I created this PR https://github.com/cilium/tetragon/pull/2231 to implement this feature request. I will be glad to hear any feedback/comments.