Bottlerocket merging bootstrap_extra_args are adding extra quotes when using the official eks terraform module

[Jinkxed]

Platform I'm building on:

Initializing provider plugins...
- Finding hashicorp/tls versions matching ">= 3.0.0"...
- Finding hashicorp/time versions matching ">= 0.9.0"...
- Finding hashicorp/cloudinit versions matching ">= 2.0.0"...
- Finding hashicorp/aws versions matching ">= 4.0.0, >= 4.33.0, >= 4.36.0, >= 4.47.0, >= 4.57.0, >= 5.0.0, ~> 5.9"...
- Finding hashicorp/kubernetes versions matching ">= 2.10.0, >= 2.20.0"...
- Finding latest version of hashicorp/external...
- Finding hashicorp/helm versions matching ">= 2.9.0"...
- Installing hashicorp/external v2.3.2...
- Installed hashicorp/external v2.3.2 (signed by HashiCorp)
- Installing hashicorp/helm v2.12.1...
- Installed hashicorp/helm v2.12.1 (signed by HashiCorp)
- Installing hashicorp/tls v4.0.5...
- Installed hashicorp/tls v4.0.5 (signed by HashiCorp)
- Installing hashicorp/time v0.10.0...
- Installed hashicorp/time v0.10.0 (signed by HashiCorp)
- Installing hashicorp/cloudinit v2.3.3...
- Installed hashicorp/cloudinit v2.3.3 (signed by HashiCorp)
- Installing hashicorp/aws v5.32.1...
- Installed hashicorp/aws v5.32.1 (signed by HashiCorp)
- Installing hashicorp/kubernetes v2.25.2...
- Installed hashicorp/kubernetes v2.25.2 (signed by HashiCorp)

Using bottlerocket in a eks_managed_node_group with spot instances. Nothing custom.

What I expected to happen: At first I thought this was a terraform issue, but the more I've dug in I think it might be a bottlerocket issue, but please let me know if I'm wrong.

I'm using the following settings:

  # EKS Managed DEFAULTS Node Groups
  eks_managed_node_group_defaults = {
    force_update_version              = true
    platform                          = "bottlerocket"
    ami_type                          = "BOTTLEROCKET_ARM_64"
    desired_size                      = 1
    min_size                          = 1
    max_size                          = 10
    launch_template_os                = "bottlerocket"

    bootstrap_extra_args = <<-EOT
      [settings.kubernetes]
      "image-gc-high-threshold-percent" = "85"
      "image-gc-low-threshold-percent" = "80"
      "max-pods" = "300"

      [settings.kubernetes.kube-reserved]
      "cpu" = "200m"
      "ephemeral-storage" = "1Gi"
      "memory" = "100Mi"

      [settings.kubernetes.system-reserved]
      "cpu" = "100m"
      "ephemeral-storage" = "1Gi"
      "memory" = "100Mi"
    EOT

    update_config = {
      # Do not delete existing nodes if new ones fail
      max_unavailable_percentage = 1 # or set `max_unavailable`
    }

    create_worker_security_group      = false
    # We are using the IRSA created below for permissions
    # This is a better practice as well so that the nodes do not have the permission,
    # only the VPC CNI addon will have the permission
    # NOTE: Turning this to false can cause issues with nodes not
    # being able to join the cluster.
    iam_role_attach_cni_policy        = true

    iam_role_additional_policies = {
      AmazonEBSCSIDriverPolicy = "arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy"
      AdditionalAccess = aws_iam_policy.additional_access.arn
    }

    # Required for Application Load balancers to access the pods directly
    vpc_security_group_ids = [data.aws_security_group.internal_network.id]

    # Bottlerock has 2 volumes by default
    # xvda is: operating system itself and contains the active & passive partitions
    # the bootloader, the dm-verity data, and the datastore for the Bottlerocket API.
    # xvdb is: container images, storage for running containers,
    # and persistent storage/volumes for containers.
    ebs_optimized               = true
    block_device_mappings = {
      xvda = {
        device_name = "/dev/xvda"
        ebs = {
          volume_size           = 10
          volume_type           = "gp3"
          encrypted             = true
          kms_key_id            = data.aws_kms_key.current.arn
          delete_on_termination = true
        }
      },
      xvdb = {
        device_name = "/dev/xvdb"
        ebs = {
          volume_size           = 100
          volume_type           = "gp3"
          encrypted             = true
          kms_key_id            = data.aws_kms_key.current.arn
          delete_on_termination = true
        }
      }
    }
  }

  # EKS Managed SPOT Node Groups
  eks_managed_node_groups = {
    "eks-arm-spot" = {
      capacity_type  = "SPOT"
      labels = {
        role = "spot"
      }

      instance_types  = local.arm_spot_cluster_instance_types
      desired_size    = local.arm_spot_cluster_count_desired
      instances_distribution = {
        on_demand_base_capacity                   = 0
        on_demand_percentage_above_base_capacity  = 25
        # Recommended by AWS for best price / capacity
        spot_allocation_strategy                  = "price-capacity-optimized"
        # This is result of 3 AZs * 8 instances per AZ
        spot_instance_pools                       = 24
      }

      lifecycle = {
        create_before_destroy = true
        ignore_changes = [
          local.arm_spot_cluster_count_desired
        ]
      }
    }
  }

What should happen is these are merged into the user_data and passed onto the image, which totally works IF I don't use the bootstrap_extra_args. The problem happens after that.

Take note I am using spot instances.

Here is an example of the user_data from inside the launch template without the bootstrap_extra_args args passed:

[settings.kubernetes]
"cluster-name" = "test-cluster"
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 110
[settings.kubernetes.node-labels]
"eks.amazonaws.com/nodegroup-image" = "ami-0d34ea6796ecf0833"
"eks.amazonaws.com/capacityType" = "SPOT"
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "8"
"eks.amazonaws.com/nodegroup" = "eks-arm-spot-20240116231525619800000001"
"role" = "spot"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-0f69ebb1047f87a70"

Works perfectly fine. But take note of the node-labels

What actually happened: This is what it looks like when those bootstrap args are passed:

settings.kubernetes.cluster-name = 'test-cluster'
settings.kubernetes.cluster-dns-ip = '172.20.0.10'
settings.kubernetes.max-pods = '300'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '15'
settings.kubernetes.node-labels.role = 'spot'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-0d34ea6796ecf0833'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'eks-arm-spot-20240116231525619800000001'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-0f69ebb1047f87a70'
settings.kubernetes.image-gc-high-threshold-percent = '85'
settings.kubernetes.image-gc-low-threshold-percent = '80'
settings.kubernetes.kube-reserved.cpu = '200m'
settings.kubernetes.kube-reserved.ephemeral-storage = '1Gi'
settings.kubernetes.kube-reserved.memory = '100Mi'
settings.kubernetes.system-reserved.cpu = '100m'
settings.kubernetes.system-reserved.ephemeral-storage = '1Gi'
settings.kubernetes.system-reserved.memory = '100Mi'

All of the ones that are merged in after the terraform template is applied I think? These:

settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '15'
settings.kubernetes.node-labels.role = 'spot'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-0d34ea6796ecf0833'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'eks-arm-spot-20240116231525619800000001'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-0f69ebb1047f87a70'

Have additional single quotes around the eks label ( except the spot one, I have no idea ).

How to reproduce the problem: It should be easily reproducible using a standard eks deployment using terraform, with spot and try to pass in additional bootstrap_extra_args.

Seriously wish there was an easier way to just increase the max-pod limit! Thank you for any help here, I'm pulling at straws!

[vyaghras]

@Jinkxed Thanks for raising the issue. We will look in to this and get back to you.

[vyaghras]

While going through the issue description, I noticed "double quotes" around the keys in

[settings.kubernetes]
"cluster-name" = "test-cluster"
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 110
[settings.kubernetes.node-labels]
"eks.amazonaws.com/nodegroup-image" = "ami-0d34ea6796ecf0833"
"eks.amazonaws.com/capacityType" = "SPOT"
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "8"
"eks.amazonaws.com/nodegroup" = "eks-arm-spot-20240116231525619800000001"
"role" = "spot"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-0f69ebb1047f87a70"

As referred here, the bootstrap_extra_args have been passed straight into user-data. Can you confirm if the double quotes are required around the keys?

[Jinkxed]

Let me confirm, but I'm pretty sure you will get TOML errors if they aren't.

[Jinkxed]

Sorry, actually just re-read your question. That's how they come in without setting anything custom in the terraform module.

I know when I add additional keys that if I don't double quote them it will give TOML errors, the documentation shows it both ways? https://registry.terraform.io/modules/terraform-aws-modules/eks/aws/18.17.0#eks-managed-node-groups

[Jinkxed]

Well that's how it was previously anyway. Just tried to a test in our dev environment.

Brand new cluster with no bootstrap_extra_args everything is double quoted.

[settings.kubernetes]
"cluster-name" = "us-east-1-dev"
"cluster-dns-ip" = "172.20.0.10"
"max-pods" = 17
[settings.kubernetes.node-labels]
"eks.amazonaws.com/nodegroup-image" = "ami-0dbfa71c674062f95"
"eks.amazonaws.com/capacityType" = "SPOT"
"eks.amazonaws.com/sourceLaunchTemplateVersion" = "1"
"eks.amazonaws.com/nodegroup" = "eks-arm-spot-2024011821273310770000001b"
"role" = "spot"
"eks.amazonaws.com/sourceLaunchTemplateId" = "lt-08bd91da980bf4d78"

If I add bootstrap_extra_args

    bootstrap_extra_args = <<-EOT
      [settings.kubernetes]
      max-pods = 300
    EOT

It comes in like this:

settings.kubernetes.cluster-name = 'us-east-1-dev'
settings.kubernetes.cluster-dns-ip = '172.20.0.10'
settings.kubernetes.max-pods = 300
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateVersion' = '2'
settings.kubernetes.node-labels.role = 'spot'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup-image' = 'ami-0dbfa71c674062f95'
settings.kubernetes.node-labels.'eks.amazonaws.com/capacityType' = 'SPOT'
settings.kubernetes.node-labels.'eks.amazonaws.com/nodegroup' = 'eks-arm-spot-2024011821273310770000001b'
settings.kubernetes.node-labels.'eks.amazonaws.com/sourceLaunchTemplateId' = 'lt-08bd91da980bf4d78'

Which fails.

[foersleo]

The flattened form you end up with has all the keys in their dotted form, where dots delimit different levels in the hierarchy. I think the extra quotes come in for most of the node-labels, because they contain dots . in the key name. If they were not quoted the . in the key names would probably be interpreted as other layers of the settings tree. I do not think that kubernetes has extra suboptions eks.amazonaws within node-labels, so the quotation there seems right and aligns with TOML's grammar.

So the quotes you have in your extra user data around the keys seem to be necessary (at least for the node-labels) so that TOML interpreters do not mistake the dots in them as delimiters for the key hierarchy. It also seems correct that your output after merging has the extra quotes around them to ensure the dots in them are not mistaken for hierarchical delimiters in the TOML.

There is one difference though between the two cases from the TOML perspective. Strings with single quotation marks ' in toml are literal strings, while Strings with double quotations marks " are "basic strings". I do not see where that would make a difference in the example provided, but I am not completely sure.

Having not reproduced the issue I am not quite clear at what step you are running into issues with this. Is the instance never coming up properly or has it issues parsing the user data? Is is bailing out before even starting the instance? Could you elaborate on what you mean with "Which fails."

[bryantbiggs]

FYI - this is not related to the EKS module; its how EKS managed nodegroups merge user data provided by users with the user data it provides. Thats where the changing to the flattened TOML is taking place

I don't now if that is causing an issue, or if there is an issue (that was unclear to me), but that is where the change in TOML form is coming from