[EKS] [request]: Create a notification for new versions of the EKS Optimized AMI

[jicowan]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
• Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
• If you are interested in working on this issue or have submitted a pull request, please leave a comment

Tell us about your request What do you want us to build? Create an SNS topic for notifying customer when a new version of the EKS Optimized AMI is available.

Which service(s) is this request for? EKS

Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard? What outcome are you trying to achieve, ultimately, and why is it hard/impossible to do right now? What is the impact of not having this problem solved? The more details you can provide, the better we'll be able to understand and solve the problem. Today, you have to call an API to upgrade the worker nodes in a managed node group. A notification would tell customer when a new version is available for upgrade.

Are you currently working around this issue? How are you currently solving this problem? Customers can monitor the GitHub repository, e.g. https://help.github.com/en/github/receiving-notifications-about-activity-on-github/watching-and-unwatching-releases-for-a-repository.

Additional context Anything else we should know?

Attachments If you think you might have additional information that you'd like to include via an attachment, please do - we'll take a look. (Remember to remove any personally-identifiable information.)

[tabern]

@jicowan do you imagine this is just a notification or also includes some sort of metadata about the status of the AMI (tests passing, etc...)?

[jicowan]

When the image has been published and is ready for consumption. If the message includes metadata, it should include the AMI version/region, and SSM path.

[arun-krishnakumar]

+1

[philm]

ECS has had this functionality since 2017 - https://aws.amazon.com/about-aws/whats-new/2017/03/introducing-notifications-for-new-amazon-ecs-optimized-ami-releases/. Can that implementation be leveraged for this need?

[jayackri]

We are waiting for this feature to automate our AMI upgrade story. After the SNS feature has been launched by EKS, is there a way to get notified only after a new version has been released globally?

[ryanpflynn]

I have worked around this by having a CloudWatch Event rule invoke a lambda function that checks the eks-optimized ami SSM Parameters and save the last value per k8s version in a Dynamo DB table.

[bit-cloner]

Until AWS enables this feature natively . How about a mini Infrastructure as code package ( pulumi) that can create

1. SNS topic
2. Daily Lambda to check for new versions
3. Post to SNS if there is a new Version If there is a enough interest I can work on this. Also would be nice to collaborate with someone

[cchostak]

For anyone interested. I made a sample Cloudformation template that can be deployed into the same region as the EKS cluster.

The template below has only two parameters required that are self explanatory:

CLUSTER => The Cluster name to search for managed node groups. (All node groups there will be updated) RATE => The rate that you want the update version call to be made.

The method is idempotent and can be executed safely:

1. If there are no AMI updates the node group will remain in the same version without disruption. 2. If there are updates, the flow will follow the update config specified and the nodes will be taken one at a time.

The usage:

1. Navigate to Cloudformation Console.
2. Upload the template attached (template.yaml).
3. Change the Cluster name and Rate as needed.
4. Deploy it.

The YAML template:

AWSTemplateFormatVersion: '2010-09-09'
Description: Lambda function to update Managed Node Groups for a given cluster based on a specific interval.
Parameters:

  CLUSTER:
    Type: String
    Default: efs
    Description: The Cluster name to search for Managed Node Groups

  REGION:
    Type: String
    Default: eu-central-1
    Description: The Cluster region

  RATE:
    Description: >
      The rate (frequency) that determines when CloudWatch Events runs the rule that
      triggers the Lambda function.
    Default: rate(10 minutes)
    AllowedValues:
      - rate(10 minutes)
      - rate(1 day)
      - rate(7 days)
      - rate(30 days)
    Type: String

Resources:

  LambdaSchedule:
    Type: "AWS::Events::Rule"
    Properties:
      Description: >
        A schedule for the Lambda function..
      ScheduleExpression: !Ref RATE
      State: ENABLED
      Targets:
        - Arn: !Sub ${UpdateFunction.Arn}
          Id: LambdaSchedule

  LambdaSchedulePermission:
    Type: "AWS::Lambda::Permission"
    Properties:
      Action: 'lambda:InvokeFunction'
      FunctionName: !Sub ${UpdateFunction.Arn}
      Principal: 'events.amazonaws.com'
      SourceArn: !Sub ${LambdaSchedule.Arn}

  UpdateFunction:
    Type: AWS::Lambda::Function
    Properties:
      Runtime: python3.8
      Role: !GetAtt LambdaExecutionRole.Arn
      Handler: index.handler
      Environment:
        Variables:
          cluster: !Ref CLUSTER
          region: !Ref REGION
      Code:
        ZipFile: |
          from __future__ import print_function
          import os
          import boto3

          client = boto3.client('eks')

          def handler(event, context):
              cluster = os.getenv("cluster", None)
              region = os.getenv("region", None)

              if cluster and region:
                  message = "Env vars are there cluster: {} region: {}!".format(cluster, region)
                  # List Node Groups inside cluster
                  response = client.list_nodegroups(
                      clusterName=cluster,
                  )
                  try:
                      for nodegroup in response['nodegroups']:
                          print("Node Group => {}".format(nodegroup))
                          response = client.update_nodegroup_version(
                              clusterName=cluster,
                              nodegroupName=nodegroup,
                          )
                          message = "Update => {}".format(response)
                  except:
                      message = "Something went badly wrong, do you have Managed Node groups in this cluster?"
              else:
                  message = "No env vars passed"

              print(message)
              return message
      Description: The function that perform the API Update call

  LambdaExecutionRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
        - Effect: Allow
          Principal:
            Service:
            - lambda.amazonaws.com
          Action:
          - sts:AssumeRole
      Path: "/"
      Policies:
      - PolicyName: root
        PolicyDocument:
          Version: '2012-10-17'
          Statement:
            - Effect: Allow
              Action: '*'
              Resource: '*'

[v4de]

Noticed the AMI notifications for ECS Optimized Linux AMI have not been working and it has been removed from documentation. Does this mean we wont be getting notifications from EKS and ECS AMI updates anymore?

https://github.com/awsdocs/amazon-ecs-developer-guide/commit/1114647d1722fdb796cea68f680b00e19439fbbd#diff-1734817e93fc2d78797cccf296458d94aee95fa6b8083b602b36d629020acfdb

doc_source/ECS-AMI-SubscribeTopic.md

[v4de]

SNS Topic for ECS linux was deprecated when ECS optimized linux 1 ami was depreacted. So only option is to query ssm for now. Guess i'm making my own Lambda to check versions and update.

https://github.com/awsdocs/amazon-ecs-developer-guide/issues/184

[amitkatyal]

We also have a requirement to notify the customers about the new EKS AMI availability so that customers can update the CF stack to update the managed node groups. By when are we planning to release this feature?

[ispirals]

We've solved the problem of getting notified when a new AMI for the clusters we have is available by creating a custom lambda that checks the current image used by the nodes and the latest image of the same Kubernetes version, sending a message to a specific SNS topic.

But now, we're trying to solve a different problem that is getting notified when a new version of EKS (Kubernetes) is available, but I'm having a lot of issues because I can't find a way to retrieve the available versions or the latest version of Kubernetes in EKS (in a programmatic way). Any suggestion here?

[Aventuz]

We've solved the problem of getting notified when a new AMI for the clusters we have is available by creating a custom lambda that checks the current image used by the nodes and the latest image of the same Kubernetes version, sending a message to a specific SNS topic.

But now, we're trying to solve a different problem that is getting notified when a new version of EKS (Kubernetes) is available, but I'm having a lot of issues because I can't find a way to retrieve the available versions or the latest version of Kubernetes in EKS (in a programmatic way). Any suggestion here?

Hi isprials, can you share the lambda code with me?

[anithreddy95]

We've solved the problem of getting notified when a new AMI for the clusters we have is available by creating a custom lambda that checks the current image used by the nodes and the latest image of the same Kubernetes version, sending a message to a specific SNS topic.

But now, we're trying to solve a different problem that is getting notified when a new version of EKS (Kubernetes) is available, but I'm having a lot of issues because I can't find a way to retrieve the available versions or the latest version of Kubernetes in EKS (in a programmatic way). Any suggestion here?

Hello isprials, can you please share the lambda code in the thread or directly with me, that would be helpful. Thanks in advance

[bgdnlp]

This issue doesn't specify, but please don't forget about Bottlerocket.

This aws-cli command will return the latest x86-64 Bottlerocket AMI for EKS 1.21 in us-east-1.

aws --region us-east-1 ssm get-parameter \
  --name /aws/service/bottlerocket/aws-k8s-1.21/x86_64/latest/image_id \
  --query "Parameter.Value"

And this oneliner should print the AMI id to stdout if and only if it hasn't seen it before. Writes it to /tmp/eks-amis the first time it gets a new id. For the three people out there who still get emails from cron jobs it can be a quick and dirty solution.

aws --region us-east-1 ssm get-parameter \
    --name /aws/service/bottlerocket/aws-k8s-1.20/x86_64/latest/image_id \
    --query "Parameter.Value" --output text \
  | grep -v -f /tmp/eks-amis \
  | tee -a /tmp/eks-amis

[ad-m-ss]

One more workaround...

AWS EKS AMI provides RSS about new releases: https://github.com/awslabs/amazon-eks-ami/releases.atom You can subscribe RSS feed eg. in Slack: https://slack.com/help/articles/218688467-Add-RSS-feeds-to-Slack

[prashil-g]

Any update on when this feature would be available?

[rohitgujral16]

This will be good to have

[singhnix]

customers eagerly waiting for this. Please expedite.

[smiller171]

customers eagerly waiting for this. Please expedite.

There's always customers waiting for everything. Thumbs up the issue and the maintainers can quantify how many ppl care about each issue.

[nan008]

We've solved the problem of getting notified when a new AMI for the clusters we have is available by creating a custom lambda that checks the current image used by the nodes and the latest image of the same Kubernetes version, sending a message to a specific SNS topic.

But now, we're trying to solve a different problem that is getting notified when a new version of EKS (Kubernetes) is available, but I'm having a lot of issues because I can't find a way to retrieve the available versions or the latest version of Kubernetes in EKS (in a programmatic way). Any suggestion here?

@ispirals can you share the lambda? that would be very helpful

[lafferrs]

We created a terraform module to solve this problem and to allow for tracking and eventing on any publicly available AMIs. It is available below.

https://github.com/Evernorth/aws-ami-tracker

[PRYeswanthReddy]

Can this be prioritized please.

[tylergohl]

One more workaround...

AWS EKS AMI provides RSS about new releases: https://github.com/awslabs/amazon-eks-ami/releases.atom You can subscribe RSS feed eg. in Slack: https://slack.com/help/articles/218688467-Add-RSS-feeds-to-Slack

This is a decent option but includes pre-release notifications as well..

[smanavi-NS1]

I am trying to get notifications regarding updates pending or new updates on AWS console. Can you please guide me how can we achieve using SNS and Lambda?