kubernetes
LoadBalancer Service Provisions in Untagged VPC Subnets
What happened:
A breaking change in the subnet auto-discovery of Service load balancer has been introduced by the PR: https://github.com/kubernetes/kubernetes/pull/97431.
Prior to this change, the AWS Cloud Controller Manager (CCM) respected user intention by only auto-discovering subnets explicitly tagged for the cluster (kubernetes.io/cluster/<id>:owned|shared) when provisioning load balancers.
Following the mentioned change, the controller now automatically discovers all subnets in the VPC, including ones without any cluster tag. This generates an issue for users who want to install multiple clusters in the same VPC that has many subnets. The change breaks the established behavior and can lead to unintended subnet usage.
As per the PR description, this was an intentional change:
Prior to this change, the AWS cloudprovider would auto discover subnets only with the cluster tags when provisioning NLB/CLB for service resources. We want to modify this behavior to include the subnets without any cluster tags, in addition to the ones previously matched by auto-discovery. After the changes, the auto-discovery will consider all subnets except the ones tagged explicitly for other cluster. If there are multiple subnets per AZ, the ties are broken in the following order
This breaking change has impacted specific scenarios, such as:
- Untagged subnets are now being used for LoadBalancer services, violating user intent and potentially causing network issues.
- Kubernetes distributions, such as OpenShift, that install a cluster in an existing VPC with a subset of subnets are impacted, particularly after a cluster upgrade.
What you expected to happen:
The AWS Cloud Controller Manager could only auto-discover and use subnets for Service LoadBalancer provisioning that are explicitly tagged with kubernetes.io/cluster/<id>:owned|shared. This is the expected behavior to ensure the controller respects the user's defined cluster boundaries and prevents the use of unintended subnets.
How to reproduce it (as minimally and precisely as possible):
- Start with an existing AWS VPC that has at least two public subnets in distinct zones, or create a subnet in distinct zone from existing one (without cluster tags).
- Tag only one of these subnets with
kubernetes.io/cluster/<cluster_name>:owned. - Deploy an OpenShift cluster into the existing VPC, ensuring it only uses the single, tagged subnet for its installation.
- Create a new service of type LoadBalancer using a simple application deployment. For example:
apiVersion: v1
kind: Service
metadata:
name: my-lb-service
spec:
type: LoadBalancer
ports:
- port: 80
targetPort: 8080
selector:
app: my-app
- Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.
Anything else we need to know?:
This issue is particularly impactful for multi-cluster environments or when deploying a single-subnet cluster within a larger, existing VPC. The lack of respect for the kubernetes.io/cluster/<id>:owned tag breaks a core assumption of how clusters are deployed in shared VPCs.
Please see the related OpenShift bug for additional context and impact reports:
- https://issues.redhat.com/browse/OCPBUGS-17432
Other relevant discussions:
- https://github.com/kubernetes/kubernetes/pull/97431#issuecomment-998989533
- https://github.com/kubernetes/cloud-provider-aws/issues/1201
Environment:
- Kubernetes version (use
kubectl version): 1.20+ (version that kubernetes/pull/97431 was backported) - Cloud provider or hardware configuration: AWS
- OS (e.g. from /etc/os-release):
- Kernel (e.g.
uname -a): - Install tools: OpenShift
- Others:
/kind bug
This issue is currently awaiting triage.
If cloud-provider-aws contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.
The triage/accepted label can be added by org members by writing /triage accepted in a comment.
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.
Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.
@mtulio Is the suggestion here that the order of precedence is wrong?
If there are multiple subnets per AZ, the ties are broken in the following order
What is the finish of this thought from the docs you copied?
@JoelSpeed :
Observe the load balancer's creation. The CCM will provision the load balancer in one of the untagged subnets, instead of the intended, tagged subnet.
@mtulio Is the suggestion here that the order of precedence is wrong?
No, the report is that after https://github.com/kubernetes/kubernetes/pull/97431 an important feature/behavior of controller had been changed (forcing discoverySubnets to use subnets outside of cluster-scope / untagged subnets), generating user's noisy and re-engineering in kubernetes distribution to workaround of the new behavior.
Considering it was introduced 10+ releases ago and now this may become de-facto a new controller behavior, I am not sure yet what to suggest, just want to bring the discussion as I didn't find the "tag standard for kubernetes.io/cluster/id" on k8s docs.
Should we need to document "well-known" tags K8s, CCMs, or some other documentation? Should we need to start increasing the scope of the e2e test scenarios here in CCM-AWS preventing any new breaking cases like that on Service loadbalancers? Would like to hear from you all suggestions.
If there are multiple subnets per AZ, the ties are broken in the following order
What is the finish of this thought from the docs you copied?
This is from the PR https://github.com/kubernetes/kubernetes/pull/97431 description:
- prefer the subnet with the correct role tag. kubernetes.io/role/elb for public and kubernetes.io/role/internal-elb for private access
- prefer the subnet with the cluster tag kubernetes.io/cluster/<Cluster Name>
- prefer the subnet that is first in lexicographic order
This PR also introduces an additional annotation [...]
The Kubernetes project currently lacks enough contributors to adequately respond to all issues.
This bot triages un-triaged issues according to the following rules:
- After 90d of inactivity,
lifecycle/staleis applied - After 30d of inactivity since
lifecycle/stalewas applied,lifecycle/rottenis applied - After 30d of inactivity since
lifecycle/rottenwas applied, the issue is closed
You can:
- Mark this issue as fresh with
/remove-lifecycle stale - Close this issue with
/close - Offer to help out with Issue Triage
Please send feedback to sig-contributor-experience at kubernetes/community.
/lifecycle stale
