(eks): support latest EKS engine version v1.22

[mburket]

Describe the bug

We just updated the CDK version from 2.20 to 2.23 and we are using the latest EKS 1.22 version. Looks like the EKS 1.22 is removed from 2.22+.

Error from npm build: "error TS2551: Property 'V1_22' does not exist on type 'typeof KubernetesVersion'. Did you mean 'V1_20'?"

Expected Behavior

EKS v1.22 should not be removed.

Current Behavior

EKS v1.22 is not supported.

Reproduction Steps

Specify Kubernete version of V1_22 when create an EKS cluster using CDK 2.22+.

Possible Solution

No response

Additional Information/Context

No response

CDK CLI Version

2.23

Framework Version

No response

Node.js Version

14

OS

Mac

Language

Typescript

Language Version

No response

Other information

No response

[robertd]

@mburket See #20000 for why v1.22 was reverted.

[kaizencc]

Echoing what @robertd mentioned above. We're working on reducing the module size to allow multiple versions of kubectl lambda layers, but at this time, we can't support v1.22.

[github-actions[bot]]

⚠️COMMENT VISIBILITY WARNING⚠️

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[sdenardi]

Hi @robertd and @kaizencc. Now that v1.21 is EOL, is there any progress or timeline update on bringing v1.22 support to the cdk?

The question came up on my team because one of the apps we use dropped support for v1.21 in a recent release. We prefer and enjoy using the cdk to manage our resources, so are hesitant to perform an update to v1.22 through the AWS Console and have our cdk code out of sync with the cluster.

We completely get the reasons for reverting support, and appreciate yall's transparency and consideration for users. Thank you for all you do.

[kaizencc]

@sdenardi We are actively working on this via the Reduce Module Size RFC. In particular, @madeline-k is working on the lambda layer aspect of the RFC that will unlock the ability to create new layers (without massively increasing package size). I don't want to commit to a timeline atm, but this is in development and hopefully we can share further updates with y'all as they arise.

[robertd]

Hey @sdenardi, @pahud and I are brainstorming on PR draft (#20596) and how to create custom kubectl layer that will match EKS cluster version instead.

[robertd]

@kaizencc Are we doing the same thing in #20596, that @madeline-k is currently investigating?

[AnitaErnszt]

@kaizencc that RFC you mentioned seems abandoned, the last comment on it was over 3 weeks ago.

I would absolutely love to put EKS forward as the IaC tool for our EKS setup (we currently using terraform for most things..), but the fact that CDK only supports the end-of-time V1.21 makes it impossible. AWS is due to release V1.23 this month, is it safe to presume we not gonna get this version in CDK anytime soon either?

Reading the comment on the roll-back, it seems like your issue is to support <= V.1.20 whilst supporting the new version. V1.18 && V1.19 are now officially deprecated and V.20 is due to be deprecated in November. Are you planning to wait until November to resolve this issue?

[shcherbak]

@kaizencc AWS EKS 1.23 is available, but we can not upgrade even 1.21 using CDK :(

[shcherbak]

starting migration to terraform next monday sick and tired of this problem

[sdenardi]

I'm not quite ready to abandon CDK, as it fits nicely with our other tooling and have served our needs well up to this point.

It is a bit frustrating to not get any updates, while at the same time getting emails from AWS with the subject line:

Install EBS CSI driver before upgrading to EKS version 1.23

[watany-dev]

I have a problem that I can't version up eks for cdk, but here is my understanding

• EKS can be started from v1.23 to v1.20.
• kubectl is in scope for one version before and after.
• lambda-layer-kubectl is currently v1.20.
• EKS v1.19 will be out of support on 2022/08/01. https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

Therefore, we believe we can adopt at least v1.22 by taking the following actions.

1. upgrade lambda-layer-kubectl to v1.21
2. deprecate aws-eks up to v1.19. Newly support v1.22.

Frankly, I don't think it is appropriate to prevent the deployment of eks clusters for the sake of downward compatibility with lambda-layer-kubectl, so either add the ability to specify the version of lambda-layer-kubectl or truncate support for the oldest version. EKS is a version or two behind k8s, and we believe that there are many users who would like to increase their EKS version as much as possible.

[WinterYukky]

I agree with @watany-dev's https://github.com/aws/aws-cdk/issues/20263#issuecomment-1213571723 . I know that I am currently blocking PR regarding the lambda layer. (because they are working on package size improvements for the lambda layer). I know that comment is a breaking and exceptional change, but I still think it is well worth it.

If we make this change, there will be a grace period before the next support expires. If RFC is merged in the meantime, it would be possible to support multiple versions of lambda-layer.

[gkaskonas]

Any ETA on this? AWS is already pushing v1.23 and we can't upgrade

[cgarvis]

Hey everyone, we are working diligently on this and will have the new lambda layer solution out in mid September, possible sooner. We'll provide an update here Monday with how we are progressing. 🙏 Thank you for being patience with us.

[msessa]

Thanks for the update @cgarvis, looking forward to the official solution.

Meanwhile, for anyone interested, this is a workaround to enable updating EKS to 1.22/1.23.

The idea is to build a kubectl/helm/awscli lambda layer at synth-time (using docker) with a version of kubectl matching the cluster.

Usage: 1- Import this construct in your app codebase 2- Provision your cluster with something like this:

const kubeVersion = KubernetesVersion.of('1.23');
const layer = new KubeClientUtilsLayer(scope, 'KubeClientUtils', {
  kubectlVersion: `${kubeVersion.version}.0`,
});
const cluster = new Cluster(this, 'Cluster', {
  version: kubeVersion,
  kubectlLayer: layer,
});

When official support is released, it should be easy to switch back to using the built-in lambda layer

[cgarvis]

Update: Having some issues with #21481 that is delaying @kaizencc. Still on track for mid September but not sooner.

[fspaniol]

Any updates on this? :(

[rinrinne]

Why don't you migrate lambda solution from layer to container image?

[shcherbak]

we are waiting for this fix from april

[misterjacko]

Any updates for us @cgarvis or @kaizencc ?

[kaizencc]

This is coming! We have a new repository set up that will soon expose assets from multiple kubectl versions (including 1.22) and @madeline-k is working on dynamically loading that asset into aws-cdk-lib when necessary.

[nilroy]

@kaizencc Thanks for the update. Can we expect it this month?

[shapirov103]

AWS CDK team: this issue prevents any customer from leveraging CDK to provision and maintain EKS clusters. We are relying on this support for CDK EKS Blueprints for example, which is used to build production ready clusters. This may be the reason why some customers (based on the comments above) are considering moving off the CDK. The issue has been pending long enough to reassess if this tooling is the right choice for EKS. I understand that it is a question of priorities and engineering work, so if there is firm ETA, please share.

[cgarvis]

We don't have a firm ETA. But we are targeting next week's release. We will know by the end of the week if we can hit that target.

[nilroy]

@cgarvis Do you know if you are able to release this next week?

[David-Tamrazov]

@cgarvis @kaizencc hey team, are there any updates for when this fix will land?

[shcherbak]

mid September -> mid November ? @cgarvis @kaizencc

[nilroy]

@cgarvis @kaizencc this issue is really biting us. Please set some ETA so that we can plan accordingly. Right now everything seems to be in air.

[jtnz]

We were just affected by an issue with cluster autoscaler that doesn't appear to be backported to old 1.21. Feeling like second class citizens because we chose to do IaC/best practices using CDK.