⚠️ Support storing Ignition user data in S3 bucket for `AWSMachinePool`

[AndiDog]

trafficstars

(Replaces https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/5148 so I can do PR stacking)

What this PR does / why we need it:

EC2 user data is insecure and very limited in size (16 KB). Use the S3 bucket storage also for machine pools. The difference to AWSMachine is that the S3 objects must stay around because a new instance could be started by the ASG at any time, and user data with a valid token must exist. Both a lifecycle policy for regular deletion, and explicit S3 object deletion requests, are used for cleanup, so that normally, only 1-3 launch template versions' user data are stored in S3. Each launch template version relates to one S3 object.

What type of PR is this?

I marked this as breaking change for now, since the default storage – in the case of AWSMachinePool + Ignition user data format – changes to IgnitionStorageTypeOptionClusterObjectStore if nothing is explicitly specified. For me, this makes sense because the AWSMachine controller already defaulted to that storage type. It's very likely that an administrator who uses Ignition user data has also configured an S3 bucket, so I see low chances of breaking someone's workflow. If not, the reconciler will log an error and not continue, and the administrator could set AWSMachinePool.spec.ignition.storageType to UnencryptedUserData in order to keep machine pool related user data stored in plaintext on each EC2 instance.

/kind feature

Which issue(s) this PR fixes:

Didn't find an existing issue. My company ran into EC2 user data size limit problems.

Special notes for your reviewer:

This change has been successfully used in the Giant Swarm fork of CAPA since July, on production clusters. I have adapted the change to the latest upstream changes, in particular the introduction of IgnitionStorageTypeOption (https://github.com/kubernetes-sigs/cluster-api-provider-aws/pull/4700). Therefore, I retested this feature using Tilt on a plain CAPI@main + CAPA@main setup, with AWSCluster + Flatcar AMI + AWSMachinePool in my manifests.

Checklist:

• [x] squashed commits
• [ ] includes documentation
• [x] includes emojis
• [x] adds unit tests
• [ ] adds or updates e2e tests

Release note:

Support storing Ignition user data in S3 bucket for `AWSMachinePool`