Update outdated default image for AWS/GCP

[guessi]

• Fix all regions have corresponding default AMI mapping.
• Refresh outdated default images for AWS & GCP.
• Change default instance type with latest T3 series.
• Support for AWS new region: ap-southeast-3, Asia Pacific (Jakarta)

AMI information comes from upstream Ubuntu Cloud Image Finder

• https://cloud-images.ubuntu.com/locator/

Or alternative, retrieve from scripts:

AWS:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Amazon.*focal.*amd64.*20220706*/{print$4" "$18}' | \
    sed 's/\>//;s/\<\/a\>//' | \
    column -t | \
    sort

GCP:

curl -fsSL https://cloud-images.ubuntu.com/locator/releasesTable | \
    awk -F '"' '/Google.*amd64.*ubuntu-2004-focal-v20220712.*/{print$4" "$16}' | \
    column -t | \
    sort

Last Update: 2022.07.16

[mtb-xt]

@bradrydzewski @tboerger is there any chance we could merge this? It would be awesome if we could have AWS SSM agent in our drone agent instances...

[ademariag]

Any chance that we could push this at one point?

[guessi]

rebase, and update to current latest images

[ademariag]

@bradrydzewski @tboerger is there any chance we could merge this?

[tboerger]

I can't merge on this repo, it's up to @bradrydzewski

[ademariag]

Is this project dead? I don't see any updates lately and lots of valuable contributions are not being accepted. @bradrydzewski could you add more maintainers perhaps from the community?

[guessi]

FYI, just update the default images available on AWS/GCP

[guessi]

Two years later, it finally got approved 🎉

For long-term consideration, it feels like we need an auto discovery mechanism for discovering new image id for specific requirement of the platform. Those image ids are now 6 month old 👶 🍼 since my last commit.

Let me know if image ids update is required before it get merged.

[guessi]

FYI, just refreshed AMI IDs and squash commits, also I added support for AWS new region: ap-southeast-3 (Asia Pacific, Jakarta).

[bradrydzewski]

@guessi thanks, can you please provide the link where you are sourcing the AMI list? For security purposes, we need to manually verify before we can merge (sorry, this is a security requirement, trust but verify ...)

[guessi]

@bradrydzewski sure, security always first !!!

Let me know where should I put the link, put it here as PR description or as comment in code?

Update: I've update the PR description for where the AMI info comes from.

[bradrydzewski]

I noticed these images are being labeled as hvm-ssd but the autoscaler does not use ssd disk types by default. Have you tested any of these new amis with default settings to ensure there is no risk of regression?

[guessi]

@bradrydzewski I personally only use SSD nowadays and I have used these image as source for a long time :-)

If this is really a concern, we can alternatively retrieve image id from awscli instead, which might be much reliable for user?

aws ssm get-parameters \
    --names /aws/service/canonical/ubuntu/server/20.04/stable/current/amd64/hvm/ebs-gp2/ami-id

From trustworthy source mentioned in Ubuntu Official Documentation[1].

or

aws ec2 describe-images \
    --owners aws-marketplace \
    --filters "Name=owner-id,Values=679593333241" "Name=name,Values=*ubuntu-focal-20.04-amd64*" \
    --query "sort_by(Images, &CreationDate)[-1:].[Name, ImageId]" \
    --region us-east-1 \
    --output text

Where "679593333241" is Canonical's ID, where you can find how to verify it is released by Canonical here[2].

Or even more better, retrieve from AWS API.

Ref:

• [1] https://ubuntu.com/server/docs/cloud-images/amazon-ec2
• [2] https://blog.gruntwork.io/7616fe46b49a

[bradrydzewski]

I have no problem with ssd or even ssd by default, I just want to make sure that if someone runs the autoscaler with default options it will work out of the box and not error. If we use images labeled hvm-ssd do we need to make any changes to volume type or will everything just work? I am asking because I do not use AWS so the answer to this is non-obvious to me.

[guessi]

Your concern is not exist, since hvm-ssd have been used for quite long time.

To verify that we can easily find existed image ids for us-east-1[1], us-west-1[2] and you will find that current images are hvm-ssd already.

AWS US-EAST-1

$ aws ec2 describe-images --region us-east-1 --image-ids "ami-43a15f3e" --output json | grep "ImageLocation" # image id for us-east-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

AWS US-WEST-1

% aws ec2 describe-images --region us-west-1 --image-ids "ami-925144f2" --output json | grep "ImageLocation" # image id for us-west-1
"ImageLocation": "099720109477/ubuntu/images/hvm-ssd/ubuntu-xenial-16.04-amd64-server-20180306",

• [1] Image ID for us-east-1: https://github.com/drone/autoscaler/blob/810c77f/drivers/amazon/util.go#L42
• [2] Image ID for us-west-1: https://github.com/drone/autoscaler/blob/810c77f/drivers/amazon/util.go#L47

** 810c77f is current master commit hash.

[bradrydzewski]

you will find that current images are hvm-ssd already.

thanks! I think this is the context I was missing

[guessi]

Just update once again after month.

[guessi]

Umm... looks like I need to resolve conflict and bump the image version again. Let me handle it.

[guessi]

Just update the the image id for AWS and GCP once again.

At the moment of I created this PR (Sep 11, 2020) the latest version available was Ubuntu Server 20.04 LTS and 2 years later, it's now 2022, Ubuntu Server 22.04 LTS was released on Apr 22, 2022.

Feel free to let me know if it is needed to update to Ubuntu Server 22.04 LTS.

[maxgruebneraeroqual]

We've run into an issue with builds that was sorted by switching to Ubuntu 22.04 AMIs.

https://community.harness.io/t/drone-autoscaler-amis-too-old-for-net-6-application/12249/5

Further, we've just got an email from AWS about the 16.04 Ubuntu AMIs being deprecated.

[tphoney]

thanks for the continued work on this PR !

solid work !!