Suggestion: Publish AWS AMIs using gp3 block devices as default

[jfrancisco0]

What I'd like:

Currently, Bottlerocket images default to creating two volumes (as explained in the README), and the AWS AMIs have these volumes set as gp2 by default. For example, ami-0b2c067c54702aea4.

I suggest setting it to gp3 instead, since it provides lower price per GB, and the new Amazon Linux version (AL2023) already sets it as default.

I understand the volume type may be overridable, depending on one's deployment scenario, but I don't see any obvious upside to maintaining gp2 as the default, so I decided to leave this suggestion here.

[ytsssun]

Hi @jfrancisco0, thanks for bringing this up. It is a very valid suggestion to make the default volume type to gp3. We have made some evaluation on gp3 in the past. While gp3 obviously provides a more cost effective way to achieve the SSD performance with better baseline performance, we did notice that the boot time with gp3 is slightly slower than with gp2.

However, it has been a while since the last time we perform that evaluation. I will work with the team to run another boot time tests and see if we see better result.

Again, thanks for bringing this to our attention, I will keep you posted with my testing.

[jfrancisco0]

Thanks for following up @ytsssun! 😄

[flaviomoringa]

Hi, we're using a lot of instances with Bottlerocket and using GP3 for the root volume would definetly lower the costs quite a bit. Is there any work being done in this @ytsssun and @yeazelm ? Is there a way for us to change the AMI to use GP3 on the root volume in an easy way already? Or even in a not so easy ways so we can test?

Thanks

[yeazelm]

You can override the volume type at run-instances but the tool you are using to launch Bottlerocket would need to be able to do this. If you are using eksctl, this is set using volumeType like this snippet:

  - name: nodenamehere
    instanceType: m7g.large
    ....
    volumeSize: 80
    volumeType: gp3
    amiFamily: Bottlerocket

This will set the data volume to 80GB and gp3. You can also use other volume types as well if you want. Does this work for your use case @flaviomoringa ?

[flaviomoringa]

@yeazelm maybe... I'm using the https://github.com/terraform-aws-modules/terraform-aws-eks module, so I will try on the eks_managed_node_groups pass the following config:

block_device_mappings = {
        xvda = {
          device_name = "/dev/xvda"
          ebs = {
            volume_size           = coalesce(node_group["disk_size"], 20)
            volume_type           = coalesce(node_group["disk_type"], "gp3")
            iops                  = 3000
            throughput            = 150
            encrypted             = true
            kms_key_id            = aws_kms_key.ebs.arn
            delete_on_termination = true
          }
        },
        xvdb = {
          device_name = "/dev/xvdb"
          ebs = {
            volume_size           = coalesce(node_group["disk_size"], 50)
            volume_type           = coalesce(node_group["disk_type"], "gp3")
            iops                  = 3000
            throughput            = 150
            encrypted             = true
            kms_key_id            = aws_kms_key.ebs.arn
            delete_on_termination = true
          }
        }
      }

What do you think? Would this work for xvda (root) and xvdb (data)?

[yeazelm]

That should work @flaviomoringa, but keep in mind that the root volume will not use any more space than the default 2GB (or 4GB for NVIDIA variants) so you should keep that to 2GB to reduce cost as well. All data is written to the second volume so you shouldn't need to worry too much about the root volume (/dev/xvda).

[flaviomoringa]

ah.. good to know @yeazelm .... but it's 2Gb or 20Gb? by default seems to be creating 20Gb.

[yeazelm]

It should be 2GB, not 20GB. For example, here is the latest aws-k8s-1.30 AMI data:

aws ec2 describe-images --image-id ami-0ba40e0032dd62bf2

{
    "Images": [
        {
            "Architecture": "x86_64",
            "CreationDate": "2024-08-22T23:59:28.000Z",
            "ImageId": "ami-0ba40e0032dd62bf2",
            "ImageLocation": "amazon/bottlerocket-aws-k8s-1.30-x86_64-v1.21.1-82691b51",
            "ImageType": "machine",
            "Public": true,
            "OwnerId": "651937483462",
            "PlatformDetails": "Linux/UNIX",
            "UsageOperation": "RunInstances",
            "State": "available",
            "BlockDeviceMappings": [
                {
                    "DeviceName": "/dev/xvda",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-0e9ae740b8a85f3b1",
                        "VolumeSize": 2,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                },
                {
                    "DeviceName": "/dev/xvdb",
                    "Ebs": {
                        "DeleteOnTermination": true,
                        "SnapshotId": "snap-03b91eded1ea9720c",
                        "VolumeSize": 20,
                        "VolumeType": "gp2",
                        "Encrypted": false
                    }
                }
            ],
            "Description": "bottlerocket-aws-k8s-1.30-x86_64-v1.21.1-82691b51",
            "EnaSupport": true,
            "Hypervisor": "xen",
            "ImageOwnerAlias": "amazon",
            "Name": "bottlerocket-aws-k8s-1.30-x86_64-v1.21.1-82691b51",
            "RootDeviceName": "/dev/xvda",
            "RootDeviceType": "ebs",
            "SriovNetSupport": "simple",
            "VirtualizationType": "hvm",
            "BootMode": "uefi-preferred",
            "DeprecationTime": "2026-08-22T23:59:28.000Z"
        }
    ]
}

Note the 2 for the root volume and 20 for the data volume.

[flaviomoringa]

You're absolutely right... 20 is the EKS default node storage size. I'm going to test and share my results here.

[flaviomoringa]

It worked very well, and I saw no noticeable performance lost when compared with before.

Therefore changing the AMI default volume configs to use GP3 for the volumes does seem a good approach... it would save us some money.

I've alse made a comment regarding this on issue 4078

Thanks

[flaviomoringa]

@yeazelm a question regarding the 2 Gb root volume (4Gb nvidia): Due to how Bottlerocket updates work, where basically the full OS is retrieved ( downloading the update into a separate partition and then rebooting the instance into that new partition), are you sure 2Gb is enough? seems to be a bit short to store an update? Any thoughts on that?

[arnaldo2792]

Hey @flaviomoringa, we try to keep the OS footprint as minimal as possible and we tend to not include more utilities in the root filesystem to be able to fit in it in a small volume. We are already looking at other features like compressing the kernel modules with ZSTD, or compressing the entire root filesystem to reduce the footprint even more :+1: .

[yeazelm]

@flaviomoringa to add to what @arnaldo2792 just said. We carve off half of that 2G volume for updates. The OS only takes up less than half of the disk space and we watch this space to ensure updates always fit in that half.

[FalkWoldmann]

Hi @koooosh, is this default also rolled out for the ecs-2 variant? At least when running this argument, I still see gp2: aws ec2 describe-images --image-id ami-0635da1cd30c75a50

[KCSesh]

@FalkWoldmann - currently gp3 has only been rolled out on our last new variant, aws-k8s-1.33 https://github.com/bottlerocket-os/bottlerocket/pull/4513

[FalkWoldmann]

Thanks for the quick reply! Is the rollout also planned for ecs?