Add EKS AutoMode support

[richardcase]

/kind feature /priority important-soon /triage accepted /help

Describe the solution you'd like

Add support for EKS Auto mode to CAPA.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

[k8s-ci-robot]

@richardcase: This request has been marked as needing help from a contributor.

Guidelines

Please ensure that the issue body includes answers to the following questions:

• Why are we solving this issue?
• To address this issue, are there any code changes? If there are code changes, what needs to be done in the code and what places can the assignee treat as reference points?
• Does this issue have zero to low barrier of entry?
• How can the assignee reach out to you for help?

For more details on the requirements of such an issue, please see here and ensure that they are met.

If this request no longer meets these requirements, the label can be removed by commenting with the /remove-help command.

In response to this:

/kind feature /priority important-soon /triage accepted /help

Describe the solution you'd like

Add support for EKS Auto mode to CAPA.

Anything else you would like to add: [Miscellaneous information that will assist in solving the issue.]

Environment:

• Cluster-api-provider-aws version:
• Kubernetes version: (use kubectl version):
• OS (e.g. from /etc/os-release):

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[richardcase]

Whoever picks this up we should think about the API changes upfront. Does it fit into AWSManagedControlPlane or do we need a new resource kind?

[seanschneeweiss]

This will require the upgrade to AWS SDK v2 in #2225. We created a fork to include EKS Auto Mode support in the v1 SDK: https://github.com/c445/aws-sdk-go/commit/309a05d2c76eb43157f57cd8b4346cd94c6b99b5 The fork of SDK v1 was just a quick starter for us. I guess for CAPA upstream the SDK v2 and the CRD change is the way forward.

IMHO it fits into AWSManagedControlPlane. On the long run, a mixed mode of EKS Auto Mode and non EKS Auto Mode might be supported by EKS.

Node Pools in EKS Auto Mode differ from Amazon EKS Managed Node Groups but can coexist in the same cluster. src

I wished this could be aligned with ACK controller for EKS, but there is no progress yet: https://github.com/aws-controllers-k8s/community/issues/2222

The AWS SDK exposes Auto Mode settings as follows:

ComputeConfig *ComputeConfigRequest

type ComputeConfigRequest struct {
	// Request to enable or disable the compute capability on your EKS Auto Mode
	// cluster. If the compute capability is enabled, EKS Auto Mode will create
	// and delete EC2 Managed Instances in your Amazon Web Services account.
	Enabled *bool `locationName:"enabled" type:"boolean"`
	// Configuration for node pools that defines the compute resources for your
	// EKS Auto Mode cluster. For more information, see EKS Auto Mode Node Pools
	// in the EKS User Guide.
	NodePools []*string `locationName:"nodePools" type:"list"`
	// The ARN of the IAM Role EKS will assign to EC2 Managed Instances in your
	// EKS Auto Mode cluster. This value cannot be changed after the compute capability
	// of EKS Auto Mode is enabled. For more information, see the IAM Reference
	// in the EKS User Guide.
	NodeRoleArn *string `locationName:"nodeRoleArn" type:"string"`
}

Hint about node pools:

nodePools: EKS Auto Mode includes general-purpose and system default Node Pools. src & docs

StorageConfig *StorageConfigRequest

type StorageConfigRequest struct {
	// Request to configure EBS Block Storage settings for your EKS Auto Mode cluster.
	BlockStorage *BlockStorage `locationName:"blockStorage" type:"structure"`
}

type BlockStorage struct {
	// Indicates if the block storage capability is enabled on your EKS Auto Mode
	// cluster. If the block storage capability is enabled, EKS Auto Mode will create
	// and delete EBS volumes in your Amazon Web Services account.
	Enabled *bool `locationName:"enabled" type:"boolean"`
}

KubernetesNetworkConfig *KubernetesNetworkConfigRequest

type KubernetesNetworkConfigRequest struct {
	// Indicates the current configuration of the load balancing capability on your
	// EKS Auto Mode cluster. For example, if the capability is enabled or disabled.
	ElasticLoadBalancing *ElasticLoadBalancing
}

type ElasticLoadBalancing struct {
	// Indicates if the load balancing capability is enabled on your EKS Auto Mode
	// cluster. If the load balancing capability is enabled, EKS Auto Mode will
	// create and delete load balancers in your Amazon Web Services account.
	Enabled *bool `locationName:"enabled" type:"boolean"`
}

API draft Draft of a potential autoMode type:

autoModeConfig:
  compute:
    enabled: true
    nodePools:
      generalPurpose: true
      system: true
      nodeRoleARN: arn:aws:iam::123456789012:role/eks-node-role
  blockStorage: 
    enabled: true
  elasticLoadBalancing: 
    enabled: true

Logic for ComputeConfig:

# ComputeConfig requires nodeRoleArn to be set, if any default NodePool is enabled.
if NodePools.GeneralPurpose || NodePools.System {
  # Ensure NodePools.NodeRoleARN is set.
}

# NodePools can be set only if ComputeConfig is enabled.
if NodePools.GeneralPurpose || NodePools.System {
  # AutoModeConfig.Compute must be enabled.
}

_{Sean Schneeweiss [email protected], Mercedes-Benz Tech Innovation GmbH, Provider Information}

[phoban01]

Happy to take a look at this.

[richardcase]

Excellent, thanks

/assign phoban01

[seanschneeweiss]

Thank you. Let me know if we can support in that direction.

[seanschneeweiss]

Auto Mode added to ACK EKS controller: https://github.com/aws-controllers-k8s/eks-controller/commit/4841113566934991ca2a6494335b7367bd262ffb#diff-dd6e562816fd92bf542bdff8797acf6c385f50a08cdb202636b48f3fe55258ca

Seems like a direct copy of the SDK structs.

[seanschneeweiss]

Completion of #5498 will add the API changes for activating Auto Mode.

I've updated my comment about the API design for AWSManagedControlPlane. I stumbled on NodeRoleARN, when not using any default node pool. So it needs to be empty if not defining any node pools upon EKS creation.

Also, the vpcCNI and Addon reconciliation requires a bit of work. There is no CNI, no DaemonSet at all when initializing the Auto Mode cluster.

[punkwalker]

#5498 is merged. @phoban01 or @seanschneeweiss you are good to go!

[seanschneeweiss]

@punkwalker that is great news. If anyone is available to take this task, please go for it. Any input on the API for Auto Mode in AWSManagedControlPlane?