containers-roadmap

containers-roadmap copied to clipboard

Tell us about your request It would be nice if it was possible to use PodNodeSelector in EKS.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? In our EKS clusters, we are looking to leverage namespaces to represent our deployment environments. In order to do this safely, we want to label different auto-scaling groups of nodes as production and non-production. We then could use have the clusterDefaultNodeSelector select the non-production nodes, and the prod namespace target production nodes.

Are you currently working around this issue? We are currently considering 2 workarounds.

1. Having our CD pipeline inject a nodeSelector into each deployment manifest based on the namespace being deployed to.
2. Writing a MutatingAdmissionWebhook to handle node selection, but this seems like we would just be re-writing the built-in PodNodeSelector.

The PodNodeSelector is an old-skool, compiled-in Admission Controller. A MutatingAdmissionWebhook is the more contemporary way to implement this functionality and requires no messing with the control plane configuration. So you are doing the right thing there I think. Plus your out-of-tree Admission Controller can be more flexible to your needs. Implementing this replacement would be good thing. Node selectors and thus PodNodeSelector should eventually be deprecated and replaced by affinity (once more efficient).

But if it were easy to enable PodNodeSelector right now, it would be a quicker fix for your issue.

Another solution I've considered using: give each auto scaling group a different taint. Say the current production environment is called "release 2019-05-01", then one ASG would have the taint release=2019-05-01:NoSchedule, and pods for that release have a toleration {"key": "release", "operator": "Equals", "value": "2019-05-01", "effect": "NoSchedule"}. For new staging releases, you'd make another ASG with a unique taint, and have its pods tolerate it. Thus, pods from the different releases couldn't intermingle on the same node.

So right now even with the latest EKS 1.13 eks.2 we don't have "PodNodeSelector" feature right?

My requirement is that to run pods on nodes on default namespace which has label lifecycle=Ec2Spot and run dashboard,grafana,prometheus etc.. (management stuff) on ASG labeled lifecycle=OnDemand so to achieve that

https://stackoverflow.com/questions/52487333/how-to-assign-a-namespace-to-certain-nodes

+1

Now that there are managed node groups in 1.14 this would be super useful to automatically restrict pods to certain groups only.

For those whom still want podnodesselector functionality but can't be bothered to write the admission webhook controllers: https://github.com/liangrog/admission-webhook-server

@tabern is there any hope that it will be activated on EKS soon?

Yes, any update?

Any update, this is no longer alpha

We're taking a look at this.

Update 11/11 We took a deeper look at this. Because the namespaces that are declared for this controller are statically defined in config files that are on the api server, it is not something that we can easily enable today. We're keeping this on our backlog and will continue to evaluate the feasibility of properly enabling this controller in the future.

Is there any other solution to this problem? We have a 3rd party helm chart that doesn't cleanly handle setting tolerations and affinity. We would need everything to run on a specific node group and this would have solved the problem

Any news on this?

There really is a need for this feature. It would be great to have it on EKS. ❤️

@tabern AKS supports this admission controller and it is an easy solution to an otherwise (more) complicated setup with custom mutation webhooks (be that OPA or something else). It solves quite an important aspect of enforcing multi-tenancy on a single cluster. Can you maybe elaborate on why the admission controller can't be made to work within the EKS context?

We would like to use this as well.

Any updates on this feature yet ? Would make life much easier if this is supported

This is a great feature, please add this to EKS.

This would be a great feature but AWS makes feature requests based on how many 👍🏻 the issue receives. To date, there are 44 that have more. If you want this to have a chance the only way is to get more 👍🏻 on the issue.

I am sure there will be more requests for this when more people and organizations start to use EKS over AWS WaveLength since you want to deploy on a specific WL and it is much easier to use PodNodeSelector at the namespace level.

Need this for EKS!!!

need this for EKS. AKS has this enabled and readily available.

We took a deeper look at this. Because the namespaces that are declared for this controller are statically defined in config files that are on the api server, it is not something that we can easily enable today. We're keeping this on our backlog and will continue to evaluate the feasibility of properly enabling this controller in the future.

Usually noone is interested in the file format configuration, most ppl are interested in the configuration annotation format. So i think the point is rather moot as that is not the primary or the only way of configuring this.

For multi tenant clusters this is a big issue, and i would prefer to not roll my own solution if i have another option that is currently builtin.

Edit: grammar.

+1

👍

+1

This will be a great feature and help us to solve some problems we are facing if enabled in EKS.

+1

+1

As iverberk mentioned, this can be achieved using Gatekeeper mutation feature. It automatically injects tolerations and nodeaffinity to pods at the deployment time. A great documentation with ready to use configuration can be found here. https://medium.com/google-cloud/automating-tenant-pod-scheduling-using-policy-controller-mutations-b325f4616943