Userdata seems not working while using AWSManagedMachinePool

[pixiono]

trafficstars

/kind bug

We are trying to build an EKS cluster with NodeGroups, using the AWSManagedMachinePool. Since we are in an almost airgapped environment (i.e. have no internet gateway or similar) we would like to configure a proxy. For this we need to do a few things on the node hostsystem and would like to upload a custom userdata script for this.

What steps did you take and what happened: Since the documentation doesn't seem to cover our use case, we took inspiration from this repo and found the following: cluster-template-eks-managed-machinepool-with-launch-template-only.yaml and replaced the variables with our values.

We also adjusted the Secret so it could be created easily:

apiVersion: v1
kind: Secret
metadata:
  name: "capi-test-pool-lt-0-userdata"
stringData:
  value: "USER_DATA"
type: Opaque

We didn't changed the value.

What did you expect to happen: We have expected that, according to source code, the user data is stored either in the AWS Systems Manager or in the AWS Parameter Storage and is taken into account when the EC2 machines are started. We also connected to the EC2 instance via SSM and tried to download the userdata (without success):

[root@ip-10-16-0-27 templates]# TOKEN=`curl -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    56  100    56    0     0  28297      0 --:--:-- --:--:-- --:--:-- 56000

[root@ip-10-16-0-27 templates]# curl -H "X-aws-ec2-metadata-token: $TOKEN" -v http://169.254.169.254/latest/user-data/
*   Trying 169.254.169.254:80...
* Connected to 169.254.169.254 (169.254.169.254) port 80 (#0)
> GET /latest/user-data/ HTTP/1.1
> Host: 169.254.169.254
> User-Agent: curl/7.79.1
> Accept: */*
> X-aws-ec2-metadata-token: <TOKEN>
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< X-Aws-Ec2-Metadata-Token-Ttl-Seconds: 21593
< Content-Type: application/octet-stream
< Accept-Ranges: none
< Last-Modified: Thu, 01 Sep 2022 13:01:45 GMT
< Content-Length: 2141
< Date: Thu, 01 Sep 2022 13:15:42 GMT
< Server: EC2ws
< Connection: close
<
MIME-Version: 1.0
Content-Type: multipart/mixed; boundary="//"

--//
Content-Type: text/x-shellscript; charset="us-ascii"
#!/bin/bash
set -ex
B64_CLUSTER_CA=< B64_CLUSTER_CA >
API_SERVER_URL=<API_SERVER_URL>
K8S_CLUSTER_DNS_IP=172.20.0.10
/etc/eks/bootstrap.sh default_capi-test-control-plane --kubelet-extra-args '--node-labels=eks.amazonaws.com/nodegroup-image=ami-01c52a64630ff492f,eks.amazonaws.com/capacityType=ON_DEMAND,eks.amazonaws.com/nodegroup=default_capi-test-pool-lt-0 --max-pods=17' --b64-cluster-ca $B64_CLUSTER_CA --apiserver-endpoint $API_SERVER_URL --dns-cluster-ip $K8S_CLUSTER_DNS_IP --use-max-pods false

Anything else you would like to add: We also tried to adjust the MachinePool like this (without any success):

apiVersion: cluster.x-k8s.io/v1beta1
kind: MachinePool
metadata:
  name: "capi-test-pool-lt-0"
spec:
  clusterName: "capi-test"
  replicas: 2
  template:
    spec:
      cloudInit:
        insecureSkipSecretsManager: false
        secureSecretsBackend: secrets-manager

Environment:

• Cluster-api-provider-aws version: v1.5.0
• Kubernetes version: (use kubectl version): v1.23.7
• OS (e.g. from /etc/os-release):

[richardcase]

For AWSManagedMachinePool we don't specify userdata, we rely on the default created userdata (which calls bootstrap.sh).

It feels like we need to expose proxy information via the API (eksconfig???) and then generate additional parts for our userdata, like this:

https://aws.amazon.com/premiumsupport/knowledge-center/eks-http-proxy-configuration-automation/

(and start using userdata for managed machine pools).

It should also be possible to use a proxy with AWSMachine and AWSMachinePool with EKS. For non-eks, it's already possible to use a proxy with CAPBK

[richardcase]

I added this to the agenda for the office hours on 5th September

[richardcase]

/triage accepted /priority important-soon /area provider/eks

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The issue has been marked as an important bug and triaged. Such issues are automatically marked as frozen when hitting the rotten state to avoid missing important bugs.

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle frozen

[k8s-triage-robot]

This issue is labeled with priority/important-soon but has not been updated in over 90 days, and should be re-triaged. Important-soon issues must be staffed and worked on either currently, or very soon, ideally in time for the next release.

You can:

• Confirm that this issue is still relevant with /triage accepted (org members only)
• Deprioritize it with /priority important-longterm or /priority backlog
• Close this issue with /close

For more details on the triage process, see https://www.kubernetes.dev/docs/guide/issue-triage/

/remove-triage accepted

[richardcase]

Adding this for future reference as it has some good examples:

https://docs.aws.amazon.com/eks/latest/userguide/launch-templates.html

We may need to extend EKSConfig or even have a new bootstrap config type.

[richardcase]

/priority important-soon

[jwitko]

/assign

[richardcase]

/lifecycle active

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten