cluster-api-provider-aws

cluster-api-provider-aws copied to clipboard

/kind bug

What steps did you take and what happened: Cluster with spec:

apiVersion: v1
kind: List
items:
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: Cluster
    metadata:
      name: aws-us-east-1-capi-eks-quickstart
      namespace: aws-us-east-1
    spec:
      clusterNetwork:
        pods:
          cidrBlocks:
            - 192.168.0.0/16
      controlPlaneRef:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: AWSManagedControlPlane
        name: aws-us-east-1-capi-eks-quickstart-control-plane
      infrastructureRef:
        apiVersion: controlplane.cluster.x-k8s.io/v1beta1
        kind: AWSManagedControlPlane
        name: aws-us-east-1-capi-eks-quickstart-control-plane
  - apiVersion: controlplane.cluster.x-k8s.io/v1beta1
    kind: AWSManagedControlPlane
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-control-plane
      namespace: aws-us-east-1
    spec:
      eksClusterName: aws-us-east-1-aws-us-east-1-capi-eks-quickstart
      addons:
        - conflictResolution: overwrite
          name: vpc-cni
          version: v1.10.1-eksbuild.1
      region: us-east-1
      sshKeyName: default
      version: v1.21.2
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-a
      namespace: aws-us-east-1
    spec:
      clusterName: aws-us-east-1-capi-eks-quickstart
      replicas: 1
      template:
        spec:
          bootstrap:
            dataSecretName: ""
          clusterName: aws-us-east-1-capi-eks-quickstart
          infrastructureRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: AWSManagedMachinePool
            name: aws-us-east-1-capi-eks-quickstart-pool-0-a
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-b
      namespace: aws-us-east-1
    spec:
      clusterName: aws-us-east-1-capi-eks-quickstart
      replicas: 1
      template:
        spec:
          bootstrap:
            dataSecretName: ""
          clusterName: aws-us-east-1-capi-eks-quickstart
          infrastructureRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: AWSManagedMachinePool
            name: aws-us-east-1-capi-eks-quickstart-pool-0-b
  - apiVersion: cluster.x-k8s.io/v1beta1
    kind: MachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-c
      namespace: aws-us-east-1
    spec:
      clusterName: aws-us-east-1-capi-eks-quickstart
      replicas: 1
      template:
        spec:
          bootstrap:
            dataSecretName: ""
          clusterName: aws-us-east-1-capi-eks-quickstart
          infrastructureRef:
            apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
            kind: AWSManagedMachinePool
            name: aws-us-east-1-capi-eks-quickstart-pool-0-c
  - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSManagedMachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-a
      namespace: aws-us-east-1
    spec:
      availabilityZones:
        - us-east-1a
      amiType: &commonAmiType AL2_ARM_64
      instanceType: &commonInstanceType t4g.nano
      diskSize: &commonDiskSize 10
      capacityType: &commonCapacityType spot
      scaling: &commonScaling
        minSize: 1
        maxSize: 3
  - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSManagedMachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-b
      namespace: aws-us-east-1
    spec:
      availabilityZones:
        - us-east-1b
      amiType: *commonAmiType
      instanceType: *commonInstanceType
      diskSize: *commonDiskSize
      capacityType: *commonCapacityType
      scaling: *commonScaling
  - apiVersion: infrastructure.cluster.x-k8s.io/v1beta1
    kind: AWSManagedMachinePool
    metadata:
      name: aws-us-east-1-capi-eks-quickstart-pool-0-c
      namespace: aws-us-east-1
    spec:
      availabilityZones:
        - us-east-1c
      amiType: *commonAmiType
      instanceType: *commonInstanceType
      diskSize: *commonDiskSize
      capacityType: *commonCapacityType
      scaling: *commonScaling

Yields the following AWSManagedControlPlane:

apiVersion: controlplane.cluster.x-k8s.io/v1beta1
kind: AWSManagedControlPlane
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"controlplane.cluster.x-k8s.io/v1beta1","kind":"AWSManagedControlPlane","metadata":{"annotations":{},"name":"aws-us-east-1-capi-eks-quickstart-control-plane","namespace":"aws-us-east-1"},"spec":{"addons":[{"conflictResolution":"overwrite","name":"vpc-cni","version":"v1.10.1-eksbuild.1"}],"eksClusterName":"aws-us-east-1-aws-us-east-1-capi-eks-quickstart","region":"us-east-1","sshKeyName":"default","version":"v1.21.2"}}
  creationTimestamp: "2021-11-26T06:21:45Z"
  finalizers:
  - awsmanagedcontrolplane.controlplane.cluster.x-k8s.io
  generation: 7
  labels:
    cluster.x-k8s.io/cluster-name: aws-us-east-1-capi-eks-quickstart
  name: aws-us-east-1-capi-eks-quickstart-control-plane
  namespace: aws-us-east-1
  ownerReferences:
  - apiVersion: cluster.x-k8s.io/v1beta1
    blockOwnerDeletion: true
    controller: true
    kind: Cluster
    name: aws-us-east-1-capi-eks-quickstart
    uid: 3e4c4e51-fbe1-4787-841a-13793972ac77
  resourceVersion: "75008"
  uid: 0a30553d-75d1-42d5-9126-c883a3882d25
spec:
  addons:
  - conflictResolution: overwrite
    name: vpc-cni
    version: v1.10.1-eksbuild.1
  associateOIDCProvider: false
  bastion:
    allowedCIDRBlocks:
    - 0.0.0.0/0
    enabled: false
  controlPlaneEndpoint:
    host: https://REDACTED.yl4.us-east-1.eks.amazonaws.com
    port: 443
  disableVPCCNI: false
  eksClusterName: aws-us-east-1-aws-us-east-1-capi-eks-quickstart
  endpointAccess: {}
  iamAuthenticatorConfig: {}
  identityRef:
    kind: AWSClusterControllerIdentity
    name: default
  network:
    cni:
      cniIngressRules:
      - description: bgp (calico)
        fromPort: 179
        protocol: tcp
        toPort: 179
      - description: IP-in-IP (calico)
        fromPort: -1
        protocol: "4"
        toPort: 65535
    subnets:
    - availabilityZone: us-east-1a
      cidrBlock: 10.0.0.0/20
      id: subnet-093cf86544bc45c4e
      isPublic: true
      natGatewayId: nat-02e6ab32d2fe8685a
      routeTableId: rtb-0a3fbf07c702ababa
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-public-us-east-1a
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: public
    - availabilityZone: us-east-1a
      cidrBlock: 10.0.64.0/18
      id: subnet-017013a5dc5cc3150
      isPublic: false
      routeTableId: rtb-06d7b76ad876860c3
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-private-us-east-1a
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/internal-elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: private
    - availabilityZone: us-east-1b
      cidrBlock: 10.0.16.0/20
      id: subnet-0d22ce3d15b58a680
      isPublic: true
      natGatewayId: nat-001d194519bb6560a
      routeTableId: rtb-0d872107a5aba495d
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-public-us-east-1b
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: public
    - availabilityZone: us-east-1b
      cidrBlock: 10.0.128.0/18
      id: subnet-056bfbf3d518ee6e5
      isPublic: false
      routeTableId: rtb-033bc345233f47d1f
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-private-us-east-1b
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/internal-elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: private
    - availabilityZone: us-east-1c
      cidrBlock: 10.0.32.0/20
      id: subnet-044cb589eef79ffa7
      isPublic: true
      natGatewayId: nat-0429f5cd8010f21e6
      routeTableId: rtb-006a68a3bff171c5e
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-public-us-east-1c
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: public
    - availabilityZone: us-east-1c
      cidrBlock: 10.0.192.0/18
      id: subnet-01dec1d2277c0cb7a
      isPublic: false
      routeTableId: rtb-0bdfc3f3344e8921e
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-subnet-private-us-east-1c
        kubernetes.io/cluster/aws-us-east-1-capi-eks-quickstart: shared
        kubernetes.io/role/internal-elb: "1"
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: private
    vpc:
      availabilityZoneSelection: Ordered
      availabilityZoneUsageLimit: 3
      cidrBlock: 10.0.0.0/16
      id: vpc-06e63e7178a89fe56
      internetGatewayId: igw-0336713b79c007ce6
      tags:
        Name: aws-us-east-1-capi-eks-quickstart-vpc
        sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
        sigs.k8s.io/cluster-api-provider-aws/role: common
  region: us-east-1
  roleName: aws-us-east-1-capi-eks-quickstart-iam-service-role
  sshKeyName: default
  tokenMethod: iam-authenticator
  version: v1.21
status:
  addons:
  - arn: arn:aws:eks:us-east-1:REDACTED:addon/aws-us-east-1-aws-us-east-1-capi-eks-quickstart/vpc-cni/febead7a-c0e8-8ce1-b6e1-be1d128c23cb
    createdAt: "2021-11-26T06:36:18Z"
    modifiedAt: "2021-11-26T06:36:20Z"
    name: vpc-cni
    status: ACTIVE
    version: v1.10.1-eksbuild.1
  conditions:
  - lastTransitionTime: "2021-11-26T06:36:20Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2021-11-26T06:24:42Z"
    status: "True"
    type: ClusterSecurityGroupsReady
  - lastTransitionTime: "2021-11-26T06:36:20Z"
    status: "True"
    type: EKSAddonsConfigured
  - lastTransitionTime: "2021-11-26T06:36:18Z"
    reason: created
    severity: Info
    status: "False"
    type: EKSControlPlaneCreating
  - lastTransitionTime: "2021-11-26T06:36:18Z"
    status: "True"
    type: EKSControlPlaneReady
  - lastTransitionTime: "2021-11-26T06:36:20Z"
    status: "True"
    type: EKSIdentityProviderConfigured
  - lastTransitionTime: "2021-11-26T06:36:20Z"
    status: "True"
    type: IAMAuthenticatorConfigured
  - lastTransitionTime: "2021-11-26T06:24:43Z"
    status: "True"
    type: IAMControlPlaneRolesReady
  - lastTransitionTime: "2021-11-26T06:22:20Z"
    status: "True"
    type: InternetGatewayReady
  - lastTransitionTime: "2021-11-26T06:24:37Z"
    status: "True"
    type: NatGatewaysReady
  - lastTransitionTime: "2021-11-26T06:24:44Z"
    status: "True"
    type: RouteTablesReady
  - lastTransitionTime: "2021-11-26T06:22:19Z"
    status: "True"
    type: SubnetsReady
  - lastTransitionTime: "2021-11-26T06:22:16Z"
    status: "True"
    type: VpcReady
  externalManagedControlPlane: true
  failureDomains:
    us-east-1a:
      controlPlane: true
    us-east-1b:
      controlPlane: true
    us-east-1c:
      controlPlane: true
  initialized: true
  networkStatus:
    securityGroups:
      bastion:
        id: sg-0fbf584bf161e2531
        ingressRule:
        - cidrBlocks:
          - 0.0.0.0/0
          description: SSH
          fromPort: 22
          protocol: tcp
          toPort: 22
        name: aws-us-east-1-capi-eks-quickstart-bastion
        tags:
          Name: aws-us-east-1-capi-eks-quickstart-bastion
          sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
          sigs.k8s.io/cluster-api-provider-aws/role: bastion
      cluster:
        id: sg-04078f0304123eb50
        name: eks-cluster-sg-aws-us-east-1-aws-us-east-1-capi-eks-quickstart-901865231
        tags:
          Name: eks-cluster-sg-aws-us-east-1-aws-us-east-1-capi-eks-quickstart-901865231
          aws:eks:cluster-name: aws-us-east-1-aws-us-east-1-capi-eks-quickstart
          kubernetes.io/cluster/aws-us-east-1-aws-us-east-1-capi-eks-quickstart: owned
      node:
        id: sg-04078f0304123eb50
        name: eks-cluster-sg-aws-us-east-1-aws-us-east-1-capi-eks-quickstart-901865231
        tags:
          Name: eks-cluster-sg-aws-us-east-1-aws-us-east-1-capi-eks-quickstart-901865231
          aws:eks:cluster-name: aws-us-east-1-aws-us-east-1-capi-eks-quickstart
          kubernetes.io/cluster/aws-us-east-1-aws-us-east-1-capi-eks-quickstart: owned
      node-eks-additional:
        id: sg-00cab6e53fd681673
        ingressRule:
        - description: SSH
          fromPort: 22
          protocol: tcp
          sourceSecurityGroupIds:
          - sg-0fbf584bf161e2531
          toPort: 22
        name: aws-us-east-1-capi-eks-quickstart-node-eks-additional
        tags:
          Name: aws-us-east-1-capi-eks-quickstart-node-eks-additional
          sigs.k8s.io/cluster-api-provider-aws/cluster/aws-us-east-1-capi-eks-quickstart: owned
          sigs.k8s.io/cluster-api-provider-aws/role: node-eks-additional
  ready: true

With the following EC2 instances:

$ aws ec2 describe-instances --region us-east-1 --output json | jq '.Reservations[].Instances[] | .InstanceId, .Placement.AvailabilityZone, .PrivateIpAddress, .PublicIpAddress'
"i-006dc1a62008c6922"
"us-east-1c"
"10.0.214.221"
null
"i-0ba45bbe9c0bf1971"
"us-east-1a"
"10.0.14.52"
"54.x.x.x"
"i-0adcffc9a3f1ce86e"
"us-east-1b"
"10.0.16.26"
"44.x.x.x"

Both subnets are being attached to the managed node group:

$ aws eks describe-nodegroup --region us-east-1 --cluster-name aws-us-east-1-aws-us-east-1-capi-eks-quickstart --nodegroup-name aws-us-east-1_aws-us-east-1-capi-eks-quickstart-pool-0-a --output json | jq '.nodegroup.subnets[]'
"subnet-093cf86544bc45c4e"
"subnet-017013a5dc5cc3150"

$ aws eks describe-nodegroup --region us-east-1 --cluster-name aws-us-east-1-aws-us-east-1-capi-eks-quickstart --nodegroup-name aws-us-east-1_aws-us-east-1-capi-eks-quickstart-pool-0-b --output json | jq '.nodegroup.subnets[]'
"subnet-0d22ce3d15b58a680"
"subnet-056bfbf3d518ee6e5"

$ aws eks describe-nodegroup --region us-east-1 --cluster-name aws-us-east-1-aws-us-east-1-capi-eks-quickstart --nodegroup-name aws-us-east-1_aws-us-east-1-capi-eks-quickstart-pool-0-c --output json | jq '.nodegroup.subnets[]'
"subnet-044cb589eef79ffa7"
"subnet-01dec1d2277c0cb7a"

What did you expect to happen: Based on #2302, I expect only the private subnet to be utilized for interface attachments.

Thank you.

Anything else you would like to add: I've tried this in two regions with similar results.

Environment:

• Cluster-api-provider-aws version: 1.1.0
• Kubernetes version: (use kubectl version): 1.21.5 (Digital Ocean) management; 1.21.2 (AWS EKS) workload
• OS (e.g. from /etc/os-release):

@matthewhembree: This issue is currently awaiting triage.

If CAPA/CAPI contributors determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

/assign

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

/remove-lifecycle rotten

/unassign /help

@richardcase: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to this:

/unassign /help

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

/remove-lifecycle stale

/milestone v2.1.0

@Ankitasw: You must be a member of the kubernetes-sigs/cluster-api-provider-aws-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Cluster API Provider AWS Maintainers and have them propose you as an additional delegate for this responsibility.

In response to this:

/milestone v2.1.0

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

Update from triage 12/2022: Every AZ has both a public and private subnet, and it's not clear that from the description of #2302 that only the private subnet should be used.

Looks like there is indeed a bug: https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/9488cb54b2db690761c3545977645611d5ea2be1/pkg/cloud/scope/shared.go#L105 should give precedence to private subnets, but currently selects subnets both private and public.

/triage needs-information /priority awaiting-more-evidence

/triage accepted

/priority important-soon

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

From office hours 2023-04-03:

• Is this a difference between AWSMachinePool and AWSManagedMachinePool fields that mean something different

/triage needs-information

/help