containers-roadmap [EKS] [request]: Allow feature gates to be set on master components

Tell us about your request I would like to be able to set feature gates on master components such as the kube-apiserver, this would allow me to enable features early.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? Right now I can't enable feature gates on any master components, while I have a particular feature gate in mind I would like the general ability to toggle feature flags upon creating/updating a cluster. I understand that the feature gates enable alpha features, so I would simply note in the documentation and a warning upon any feature flag being set that using this feature can make your cluster unstable and all feature gates set should be throughly tested before releasing to production.

The particular feature gate is the VolumeSnapshotDataSource which is needed for the EBS CSI Driver's snapshotting feature.

Are you currently working around this issue? I will deploy my own container to do the snapshotting myself.

Oct 08 '19 15:10 jammerful

@jammerful hey I'm working on similar issue.... would you be able to explain your workaround a little more in-depth? What container are you deploying?

Nov 22 '19 15:11 lanefu

@lanefu I'm assuming you want the volume snapshot feature, if you're trying to enable a random feature gate I don't think you can do that. For volume snapshots, you can use the now deprecated external storage code located here https://github.com/kubernetes-incubator/external-storage/tree/master/snapshot. I've been using it for the past few weeks, and it's working well. In the repo you will find the manifest to deploy and instructions on how to use the CRDs. Lastly, snapshotting in the CSI is beta in 1.17, so maybe by next spring/summer will be available on EKS.

Nov 22 '19 16:11 jammerful

Another good feature gate to be able to enable is VolumeSubpathEnvExpansion which is in GA for k8s v1.17 but in alpha for k8s v1.14

Jan 06 '20 13:01 rinrailin

Shouldn't the label be EKS rather than ECS?

Jan 09 '20 19:01 dgarbus

@ellenthsu Please remove the ECS label and add the EKS label.

Jan 10 '20 01:01 jammerful

Apparently alpha feature gates are not enabled on the stable channels of AKS or GKE, but they do have unstable channels that enable beta feature gates. Given that I don't know if AWS would consider doing this for production clusters, wish they would speak up.

Jan 10 '20 01:01 jammerful

@ellenthsu Please remove the ECS label and add the EKS label.

done

Jan 10 '20 01:01 ellenthsu

I'm wanting to enable the ProcMountType feature gate in order to run Docker rootless, without having to use privileged pods.

Mar 09 '20 12:03 chris-vest

I would like this feature to enable ServerSideApply

Mar 21 '20 15:03 tokiwong

We're looking for ServiceNodeExclusion since at scale, we can hit an AWS NLB service limit of 500 targets. The services we scale will not be used via the NLB (they're queue-based) so they just fill up NLB slots for no reason.

Apr 15 '20 18:04 spanky-medal

My cx is looking to enable SCTP and it requires the SCTPSupport feature gate

https://kubernetes.io/docs/concepts/services-networking/network-policies/#sctp-support

Jun 03 '20 13:06 themish95

Need to support SCTP for our telco apps too.

Jun 09 '20 21:06 hymgg

I am interested in this to enable Service Topology in EKS 1.17. I am following an example to reduce cross availability zone traffic (https://v1-17.docs.kubernetes.io/docs/concepts/services-networking/service-topology/#prefer-node-local-zonal-then-regional-endpoints) which require feature-flags to be accessible: https://v1-17.docs.kubernetes.io/docs/tasks/administer-cluster/enabling-service-topology/#enable-service-topology

Jul 26 '20 00:07 mikeaorlando

@mikeaorlando something that I discovered.....

If you're using terraform modules, it's easy to switch from EKS managed nodegroups to "worker groups". the level of config effort is about the same in terraform whether managed or unmanaged, and you can easily pass arguments for feature flags.. EX:

workers_group_defaults = {
  instance_type = "t3a.xlarge"
  key_name = "secret_key"
  kubelet_extra_args = "--feature-gates=ExpandCSIVolumes=true,ExpandPersistentVolumes=true"
}

Jul 26 '20 01:07 lanefu

@mikeaorlando @lanefu i am also trying to use Service Topology running on EKS 1.17, i created the feature gates as @lanefu described, how can i make sure the feature is enabled? @mikeaorlando did you succeeded using the feature in EKS at all? cause every time i create/edit service with the TopologyKeys it seems to be deleted after saving. any advice guys?

Jul 27 '20 14:07 DanOfir

@mikeaorlando @DanOfir I am in the same boat as you wanting to use Service Topology on EKS 1.17 and find that since it has to be enabled at the API server label I'm unable to set that feature gate. We are wanting to use https://kubernetes.io/docs/concepts/services-networking/service-topology/#only-node-local-endpoints for host binding a service on only the nodes running the pods for that service.

Jul 27 '20 22:07 chasmosis

I have been unable to use Service Topology running on EKS 1.17. My current workaround is to have all nodes in the cluster in one availability zone. This does not constrict traffic to be node-local, but reduces ingress and egress charges from cross-az traffic. My current plan is to wait until EKS 1.18.

This was helpful in determining which feature flags were set in the api-server: https://docs.aws.amazon.com/eks/latest/userguide/api-server-flags.html

Jul 28 '20 22:07 mikeaorlando

In our case, we would want to set --feature-gates=IAMIdentityMappingCRD

Aug 07 '20 12:08 3oris

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

Nov 10 '20 19:11 nicolai86

Would it be possible to have a generic way to enable any feature-gates in EKS?

I'm on 1.18 and want StartupProbes

Nov 11 '20 19:11 mbelang

In our case we want to set --feature-gates=EvenPodsSpread on EKS 1.18...

Wait, AWS put out release notes specifically saying that pod topologySpreadConstraints were supported, but did not enable the feature gate? https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Pod Topology Spread has reached beta status. You can use topology spread constraints to control how pods are spread across your cluster among failure-domains such as Regions, zones, nodes, and other user-defined topology domains. This can help to achieve high availability as well as efficient resource utilization. For more information, see Pod Topology Spread Constraints in the Kubernetes documentation.

Dec 03 '20 22:12 llamahunter

@llamahunter it seems beta features are enabled by default in k8s, so the feature gate doesn't need to be passed explicitly anymore.

Dec 03 '20 22:12 nicolai86

@nicolai86 Have you successfully set pod topologySpreadConstraints using topology.kubernetes.io/zone topologyKeys in EKS v1.18?

Dec 03 '20 23:12 llamahunter

Base on the documentation the current version supported is v1.18.9 and there is this bug that doesn't let you to get the results you are expecting from the topologySpreadConstraints. The fix was pushed to the v1.18.11 as per the release notes https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG/CHANGELOG-1.18.md, so there is nothing to do but wait until Amazon release support for v1.18.11.

Jan 15 '21 17:01 felixep

Wait, AWS put out release notes specifically saying that pod topologySpreadConstraints were supported, but did not enable the feature gate?

FYI, pod topology spread was removed from EKS docs in https://github.com/awsdocs/amazon-eks-user-guide/commit/fc59dad48cdc2258aa2070709218fde4a38d62bd. So the docs are technically accurate now (they don't claim it's supported).