[EKS] [request]: Support for Kubernetes `1.31`

[stevehipwell]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request I'd like EKS to support Kubernetes 1.31.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? I'd like to be able to use Kubernetes 1.31 on EKS and be aware of the release timeline to help plan for it.

Are you currently working around this issue? N/A

Additional context N/A

Attachments N/A

[ghost]

Thumbs up for release timeline visibility - but also "it's ready when it's ready" - even up stream kube rolls this way. Never the same date.

Thumbs down for asking for the release, they'll get it out in 3-4 weeks like they consistently have year in year out. You can prepare based on that - though not sure what you're preparing for - it's usually a walk in the park.

[stevehipwell]

Thumbs up for release timeline visibility - but also "it's ready when it's ready" - even up stream kube rolls this way. Never the same date.

@fnzwex I'm not sure which upstream you're following, Kubernetes has fixed release dates. As part of the Kubernetes release planning an actual release date to the day is provided. Sometimes this changes but it's always well communicated and open.

Thumbs down for asking for the release, they'll get it out in 3-4 weeks like they consistently have year in year out.

That's demonstrably not true, the last few releases have been prompt but there have been times when EKS has been a long way behind upstream. How is wanting to track a release a bad thing anyway?

You can prepare based on that - though not sure what you're preparing for - it's usually a walk in the park.

I can't prepare without information and generally none is forthcoming unless someone creates an issue. The release calendar isn't ever updated even with a rough estimate. The ease of upgrade, which should be a non-event, isn't important; planning is about the communication with a complex organisation.

[ascopes]

:+1: for this, on the basis that if an organisation is unable to practise releases outside of release windows, it can make schedules very tight when having to upgrade EKS every 6 months, so being able to plan in advance is very important for things like this so that work and resources can be allocated in the correct places.

This becomes even tighter when also depending on other thirdparty projects that provide support for EKS, as they also need time to update and prepare anything they need to become compatible. Technically this may be trivial but in large organisations, there is often the requirement to be able to provide evidence of compatibility prior to upgrading (which makes being able to schedule this kind of thing much more critical for many companies using AWS).

Look forward to hearing updates on this!

[MikeTomlin19]

Thumbs down for asking for the release, they'll get it out in 3-4 weeks like they consistently have year in year out. You can prepare based on that

[nicc777]

So August came and went... So much for predictable release cycles.

[ghost]

It was released upstream on the 13th of August, you're 1-3 weeks premature with your complaint.

On Mon, 2 Sept 2024 at 19:34, Nico Coetzee @.***> wrote:

So August came and went... So much for predictable release cycles.

— Reply to this email directly, view it on GitHub https://github.com/aws/containers-roadmap/issues/2410#issuecomment-2324020602, or unsubscribe https://github.com/notifications/unsubscribe-auth/APJDNWXCHA4LSEJXW6TOCBLZUQIIVAVCNFSM6AAAAABMOT4HMCVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDGMRUGAZDANRQGI . You are receiving this because you were mentioned.Message ID: @.***>

[rudimk]

Just wanted to bump this up - a rough release date/week would be great. Like others have pointed out, it tends to get tricky planning upgrades in a complex org(or even with clusters spread out across multiple orgs). Selfishly, also helps to know how much of my end-of-year break will be taken up by upgrades. Any info here would be useful, thanks so much!

[nicc777]

@rudimk - exactly! We postponed a major project release thinking we would first do the EKS upgrade and then our internal apps in order to avoid to many changes at once.

Also, @fnzwex can we start to complain for real already? It's been 3 weeks after my post and well over a month after the official Kubernetes release.

[ascopes]

Can we start to complain for real already?

Given this is a paid product from Amazon, they need to provide clear timeframes for upgrades or invest in providing (edit: free as in beer) long term support releases instead. Otherwise for any business larger than a startup, the overhead of coordinating this sort of thing across numerous teams begins to outweigh the benefits of using a managed solution rather than bringing your own Kubernetes distribution.

Transparency would be greatly appreciated on matters such as this. Even if it is just an update that it is in progress, but blocked internally.

Edit: aware there is extended support, but that is an additional cost, and it is somewhat unfair to impose extra costs on consumers due to a lack of communication on Amazon's side with regards to the update cycle.

[fredcooke]

@rudimk - exactly! We postponed a major project release thinking we would first do the EKS upgrade and then our internal apps in order to avoid to many changes at once.

This is crazy, you should be deploying frequently and the two schedules should be basically independent. Why on earth would you postpone software changes for an upgrade that can be fully performed in about 45 minutes taking your time and doing it as slow and gently as possible?

Old @fnzwex is dead, ping me instead @nicc777

Can you start to complain? Sure, but now is the earliest I would have expected it - maybe give them a week to possibly drop it and THEN complain? That'd be pretty reasonable.

@ascopes check out the support page, they do have a kind of LTS setup already, versions 1.23 through 1.27 are on "extended support" which just means "more fees for being bad at keeping up to date" - 1.30, 1.29, and 1.28 are all on standard support. See here: https://docs.aws.amazon.com/eks/latest/userguide/kubernetes-versions.html

[nicc777]

@fredcooke this is not about what is technically possible. As @rudimk mention, it's about planning. Also, no EKS upgrade we did ever took <45 minutes. Our average (historically) is about 4 hours.

[fredcooke]

@nicc777 how is that even possible? I'd be very interested in the details that make it take that long. If you want some help to cut that down to a more reasonable number let me know - available from 18th of October :-)

[nicc777]

@fredcooke - we have dozens of open support cases on various issues and our TAM is on them. This schedule slip is just an annoyance as it messes up planning.

[rudimk]

Old @fnzwex is dead, ping me instead @nicc777

won't deny, did a double-take on this

[rudimk]

Some thoughts here:

1. I don't think it was too premature to flag this three weeks ago, especially given that this isn't the first time we're seeing a delay with EKS' version rollout. I manage hundreds of clusters spread across EKS, AKS and GKE. Both Azure and Google rolled out 1.31 by mid-August(GKE did so on the 20th of August) and while I'm not sure about Microsoft's date, fairly certain it happened by the last week of August. I'm sure testing a new upstream release, ensuring one's own system components(CNI, CSI and so on) are compatible and so on take up a lot of effort and time; but clearly other hyperscalers are able to do it.
2. Customers - especially people who use Kubernetes behind abstraction layers like PaaSs are not as savvy as we might be. Upgrades genuinely make them concerned for the stability of their workloads. Being able to communicate a timeline around these upgrades is crucial to maintaining trust.
3. Like @nicc777 mentioned - yes, upgrades can take up to 4 hours. I've seen an upgrade go on for 5.5 hours. If a cluster's big enough with hundreds of nodes, a rolling instance refresh is the best way to go about it, so yes - it does take time. And it's time well-spent, but it's also time that needs to be planned for.
4. Finally - I wholly agree with @ascopes - this is a paid product. In all my years working with EKS, I'm honestly still yet to see the actual value that $70 management fee gives me. AWS has never been able to actually diagnose breaking issues with their control plane whenever I've faced them; it's always been up to me to set up control plane logs and figure my way through. Okay, big deal, fine. Happy to do it. But a modicum of comms from AWS about when they intend to push out a new version would honestly be greatly appreciated - I guess I'm just not able to see why it's so hard to just put an approximate date/week/range on that version calendar.

Apologies for the long rant, but the frustration is real, sadly.

[dims]

FYI

• https://endoflife.date/azure-kubernetes-service
• https://endoflife.date/google-kubernetes-engine

[ascopes]

@rudimk ... yes, upgrades can take up to 4 hours. I've seen an upgrade go on for 5.5 hours

This also doesn't take into account internal organizational release cadence within the businesses of consumers, nor time needed to verify that none of the internal combinations of CNIs, autoscalers, etc introduce non-functional performance regressions that could pose a business risk.

Outside of places that practise true CI/CD through to production, you can still be looking at several weeks of preparation for a large scale upgrade... it isn't always something that can just go into production on the afternoon that AWS releases it. Of course, the ball cannot get rolling until it is available from AWS, so this begins to put more time pressure on the consumer.

In an ideal world this would not be a factor, of course, but driving a whole organisational change in release paradigm just due to lack of details or time frames is a massive ask for the sake of a single product offering, and is probably overkill if it can be avoided!

[stevehipwell]

I think EKS is a great product; but the lack of transparency over releases has always been one of the few weak spots.

I can't see a downside to updating the release calendar with upstream releases, at least once they go GA, even if it's only an expected month. I also don't see why the status of the engineering can't be tracked in the GH issues like this, it'd show why we pay AWS to run our control plane. For example an EKS release (I think it was 1.18) took a long time due to an etcd integration issue, I'm glad AWS dealt with that rather than me; having this reported up front would have made me happy that AWS was taking the time to get the release right rather than frustrated that the new features weren't available.

[philslab-ninja]

In the past the release of updated Bottlerocket and EKS optimized AMIs were quite good indications

EKS 1.31 variants of Bottlerocket and the EKS optimized AMI have been releasesed recently: Bottlerocket EKS optimized AMI

I guess (!) EKS 1.31 will be released till end of this month. But thats just a guess based on past experiences of course.

[nicc777]

The EKS versions page have also been updated. Finally we can start our upgrades in sandbox environments to see what breaks (as soon as it is actually available) :-)

[Indresh-Prakash]

Could notice end of standard support as September 2025 for 1.31. Previous versions used to have 14 months duration

[nicc777]

Well spotted @Indresh-Prakash - I'm guessing the date will be updated as soon as it is released fully. The 14 month period is still quoted on the FAQ. (screenshot)

[amedirr]

Kubernetes version 1.31 is now GA on EKS. https://aws.amazon.com/about-aws/whats-new/2024/09/amazon-eks-distro-kubernetes-version-1-31/

[jennerm]

This is great news, thanks for getting it done. Looks like we need EKS Pod Identities to catch up and we'll be ready to upgrade:

Error: creating AWS EKS (Elastic Kubernetes) Pod Identity Association (<unknown>): operation error EKS: CreatePodIdentityAssociation, https response error StatusCode: 400, RequestID: e9a1f290-db0a-444e-9f32-593f2124238f, InvalidRequestException: The supported Kubernetes version is from 1.24 to 1.30.

[mikestef9]

We are aware of pod identity issue and working on fix asap

[Smana]

@mikestef9 could you please tell us where we can track this issue (eks pod identity) please? thanks

[dims]

@Smana @jennerm please try now.

[Smana]

@dims I confirm that works now, thanks

[jennerm]

Works for me too now, thanks for the quick fix.

[joshuabaird]

@mikestef9 Is there a particular version of the pod identity agent that we need to be running for v1.31 support? The agent's public repo doesn't contain any relevant commits or releases that indicate support for v1.31 was added.