old_issues_repo
old_issues_repo copied to clipboard
istio
Don't run istio-init as privileged
I noticed that the injected istio-init InitContainer in Kubernetes runs as privileged and also adds the NET_ADMIN capability. privileged basically does 3 things:
- Add all capabilities (so adding
NET_ADMINis redundant) - Run without a seccomp profile (i.e.
unconfined) - this is the default in Kubernetes anyway - Run without apparmor (only applies to AppArmor enabled systems - I imagine there is an equivalent for SELinux systems). This can be achieved (1.9+ clusters) with an annotation:
container.apparmor.security.beta.kubernetes.io/istio-init: unconfined
I recommend removing the privileged bit. If the istio-init container doesn't work without privileged, it probably means apparmor is interfering. In that case just add the unconfined annotation.
istioctl version Version: 0.4.0 GitRevision: 24089ea97c8d244493c93b499a666ddf4010b547 GitBranch: 6401744b90b43901b2aa4a8bced33c7bd54ffc13 User: root@cc5c34bbd1ee GolangVersion: go1.9.1
Thank you for the analysis. Unfortunately I don't have a 1.9 cluster to test this, I agree we should get rid of privileged asap if we could. It is ridiculous to ask developers to have this level of security to deploy a micro service into Istio.
It's not quite that bad, since it's only run as an InitContainer, but it does make it a bit harder to use with a PodSecurityPolicy.
@tallclair true it is not as bad as the istio proxy itself, which btw, we just switched to non debug proxy as the default as the debug proxy requires privileged.
This is only for init container and dies after the init, however, don't the user has to have permission to deploy privileged container in the first place to enable init container to run?
This is only for init container and dies after the init, however, don't the user has to have permission to deploy privileged container in the first place to enable init container to run?
Yes, that's what I meant by "it does make it a bit harder to use with a PodSecurityPolicy" (PodSecurityPolicy is the control that restricts whether a user can create a privileged container). I did file an issue to address this usecase, feel free to chime in there: https://github.com/kubernetes/kubernetes/issues/55435
NET_ADMIN is the one we really want, the full privileged was said to be necessary in some environment (which git blame may be able to track)
I'd really like to avoid introducing more complexity into PodSecurityPolicy, if possible: https://github.com/kubernetes/kubernetes/issues/55435
How hard would it be for the privileged initialization (iptables, presumably?) to be performed by a host agent (DaemonSet in Kubernetes), similar to kube-proxy?
NET_CAP_ADMIN is sufficient on GKE for the InitContainer to program iptables. The proxy container doesn't require anything special. privileged was added to unblock depoyment on OpenShift w/SELINUX (see https://github.com/istio/issues/issues/34#issuecomment-308905920).
Does NET_CAP_ADMIN alone conflict with PodSecurityPolicy?
Host agent is a possibility. We'd need to take care to (a) only program iptables for pods that have have the sidecar injected, (b) do it in the correct network namespace to avoid conflicts with kube-proxy rules, and (c) program iptables at the correct time to avoid inbound traffic leaking into the pod before redirects rules are programmed.
ref: https://github.com/istio/issues/issues/34, https://github.com/istio/istio/issues/1435
NET_CAP_ADMIN is also sufficient on IBM Bluemix container service to program iptables. cc @dcberg
At the moment, I think we should change the release version to remove privileged. For platforms that requires privileged, debug config is always there and can work.
Please include openshift folks before removing privileged flag.
On Fri, Jan 26, 2018, 6:54 PM Lin Sun [email protected] wrote:
NET_CAP_ADMIN is also sufficient on IBM Bluemix container service to program iptables. cc @dcberg https://github.com/dcberg
At the moment, I think we should change the release version to remove privileged. For platforms that requires privileged, debug config is always there and can work.
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/istio/issues/issues/172#issuecomment-360954539, or mute the thread https://github.com/notifications/unsubscribe-auth/AJGIxk7JfX5VDmmlaA5u04H2mCcBSHRMks5tOo_xgaJpZM4RtwP0 .
Does NET_CAP_ADMIN alone conflict with PodSecurityPolicy?
Yes, pod security policy (when we turn it on) would block NET_CAP_ADMIN by default for almost all users.
Run without a seccomp profile (i.e. unconfined) - this is the default in Kubernetes anyway Run without apparmor (only applies to AppArmor enabled systems - I imagine there is an equivalent for SELinux systems). This can be achieved (1.9+ clusters) with an annotation: Grant NET_CAP_ADMIN
All of these are terrible things to disable in order to accomplish something real :) Giving NET_CAP_ADMIN is almost equivalent to root in the host - I don't think I'd have to think very hard to find an exploit path. Turning these off are basically almost as bad as having privileged.
I don't want to completely derail istio, but if the outcome is "every user workload on the cluster runs a privileged" that's bad. That said, I don't see a reason why the injected container can't grant NET_CAP_ADMIN in the short term (add it to your cap adds) and we can look at tweaking the selinux policy by default.
(copied from the other thread)
I'm going to go and get more info from the kernel teams about whether we could safely grant NET_CAP_ADMIN in a network namespace and not have it be root on the host via some trivial path. If so, we may be able to relax that restriction from SELinux in the default kernel policy. No promises yet.
@smarterclayton - I agree with what you're saying, but I still think granting the individual privileges is prefered to the privileged catch-all. Even if it's effectively equivalent from a security perspective, explicitly listing the privileges makes it more clear what is actually being used by the container, which in turn makes it easier to remove or further restrict those privileges in the future.
I completely agree with @smarterclayton. We should be able to set NET_CAP_ADMIN without having to grant privileged for all pod deployments within the service mesh.
just to recap, only 1 init container runs with NET_CAP_ADMIN during initialization not at all 'every user workload on the cluster runs a privileged'
I'd like to propose a potential solution to this problem.
We could define a CRD that describes how connections should be routed in a pod. These rules would be expressed in a technology agnostic manner, so that they could be implemented not only with iptables but with other means. This CRD would also have a pod selector to specify which pods should get those rule. The istio control plane would create these CRD objects. The cni driver would honor them when the pod is created, but before any container is started.
I believe this way no specific privileges would be required by the pod. Also this approach is probably general enough that other service mesh implementations could leverage it.
It also seems like istioctl inject defaults to EnableCoreDump = true,
which also requires privileged = true, so you'd need to pass --coreDump false.
(Not sure if that's also an issue when using initializers for injection)
Yah. Can you open an issue to track this?
On Thu, Feb 15, 2018 at 3:01 AM, Steffen Butzer [email protected] wrote:
It also seems like istioctl inject defaults to EnableCoreDump = true, which also requires privileged = true, so you'd need to pass --coreDump false. (Not sure if that's also an issue when using initializers for injection)
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/istio/issues/issues/172#issuecomment-365878829, or mute the thread https://github.com/notifications/unsubscribe-auth/AADP0TrJKnbd7a-nT43kwpz8om4vSb1Kks5tVABpgaJpZM4RtwP0 .
-- Christian Posta twitter: @christianposta http://blog.christianposta.com http://www.christianposta.com/blog
@smarterclayton What I would really like to see from openshift is the possibility for the openshift admins to add specific init containers (images) that are allowed to run with NET_CAP_ADMIN without granting the RBAC users in the project/namespace access to running other containers with NET_CAP_ADMIN.
