[Feature request]: Declarative defaults for subnets selected when there are multiple qualified

[devopsjourney1]

Describe the feature you are requesting

The ability the set default subnets that AWS Loadbalancer chooses, when there are multiple subnets to choose from.

Motivation

We have EKS clusters that span multiple subnets for different functions. e.g. Application, Database, Proxies etc. Most of our services are configured with an annotation, so they choose the correct subnet, but occasionally a service slips by without this annotation. This causes AWS loadbalancer controller to choose a subnet for the service in less then desirable fashion, as described here: https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/57472c44170f0f75df67bb5b6e83b75a2db03231/docs/deploy/subnet_discovery.md?plain=1#L5C63-L6C25

Describe the proposed solution you'd like

Make it so we can configure a flag for "default" subnets that it selects. e.g. if you have the following subnets

subnet-a subnet-b subnet-c subnet-d

you could choose subnet-c, subnet-d to be assigned as the subnets. Applications that would like to use subnet-a and subnet-b could do so by using the aws-load-balancer-subnets annotation

Describe alternatives you've considered

If this feature cannot be created, likely we will look into enforcing all service objects to declare the aws-load-balancer-subnets annotation via a policy tool like Kyverno

Contribution Intention (Optional)

-[ ] Yes, I am willing to contribute a PR to implement this feature -[x] No, I cannot work on a PR at this time

[shraddhabang]

@devopsjourney1 Hey thanks for reaching out to us. I think this is a fair ask. Any community contributions are welcome for this.

[1mwataru]

@shraddhabang (cc @devopsjourney1)

I'm planning to work on this Issue.

Currently, I'm thinking of adding a --default-subnets option where we can specify desired subnets in an array to meet this requirement.

I expect that this option may receive both public and private subnets, or sometimes just one AZ's subnets. For these cases, I'm currently considering the following behavior:

• Pick the appropriate subnets from --default-subnets based on whether the ELB scheme is internet-facing or internal

  • For internet-facing, select public subnets from those specified; for internal, pick private subnets

• If only one AZ's subnets remains after the above selection, pick the remaining required subnets from tagged subnets in lexicographical order (same as current behavior)

Please let me know if you have any thoughts on the proposed behavior or if you have any better suggestions. Also, could you please assign this to me?

[devopsjourney1]

@shraddhabang (cc @devopsjourney1)

I'm planning to work on this Issue.

Currently, I'm thinking of adding a --default-subnets option where we can specify desired subnets in an array to meet this requirement.

I expect that this option may receive both public and private subnets, or sometimes just one AZ's subnets. For these cases, I'm currently considering the following behavior:

• Pick the appropriate subnets from --default-subnets based on whether the ELB scheme is internet-facing or internal

  • For internet-facing, select public subnets from those specified; for internal, pick private subnets

• If only one AZ's subnets remains after the above selection, pick the remaining required subnets from tagged subnets in lexicographical order (same as current behavior)

Please let me know if you have any thoughts on the proposed behavior or if you have any better suggestions. Also, could you please assign this to me?

This behavior would work for me. Thank you.

[1mwataru]

/assign

[azagarelz]

@1mwataru would the logic for distinguishing between private and public subnets still be based on the role tag?

If yes, did you instead consider providing different arguments for private and public subnets so that the subnet discovery can be completely skipped?

[1mwataru]

@azagarelz

  Sorry for the delayed response.  

If yes, did you instead consider providing different arguments for private and public subnets so that the subnet discovery can be completely skipped?  

No, my implementation does not consider whether subnets are private or public, such as by using role tags. The subnet that appears first in the order specified in --default-subnets is simply preferred for selection.   https://github.com/kubernetes-sigs/aws-load-balancer-controller/blob/f140035ccf248c7a1adad8f1612fe489a08b3a10/pkg/networking/subnet_resolver.go#L467-L514   Whether subnet discovery is applied or skipped, the above logic is executed after the subnets are filtered by such as reachability/tags.