[Bug] - Errors during downloading metadata for repository 'amazonlinux'

[talawahtech]

Describe the bug When I use CloudFormation/cloud-init to install packages on an AL2022 AMI, I frequently encounter errors.

To Reproduce Steps to reproduce the behavior:

1. Create CF template where the UserData/cloud-config section specifies the list of packages to install e.g.

  packages:
    - wget
    - git
    - gcc
    - make
    - perf
    - python

2. Launch an instance using ami-04a0a1bfbe6cb2189
3. After the instance is created, review logs using journalctl
4. See error:

cloud-init[4858]: Amazon Linux 2022 repository                    0.0  B/s |   0  B     03:02
cloud-init[4858]: Errors during downloading metadata for repository 'amazonlinux':
cloud-init[4858]:   - Curl error (28): Timeout was reached for https://al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com/core/mirrors/2022.0.20220531/x86_64/mirror.list [Failed to connect to al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com port 443 after 15201 ms: Connection timed out]
cloud-init[4858]:   - Curl error (28): Timeout was reached for https://al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com/core/mirrors/2022.0.20220531/x86_64/mirror.list [Failed to connect to al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com port 443 after 15202 ms: Connection timed out]
cloud-init[4858]:   - Curl error (28): Timeout was reached for https://al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com/core/mirrors/2022.0.20220531/x86_64/mirror.list [Failed to connect to al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com port 443 after 15200 ms: Connection timed out]
cloud-init[4858]: Error: Failed to download metadata for repo 'amazonlinux': Cannot prepare internal mirrorlist: Curl error (28): Timeout was reached for https://al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com/core/mirrors/2022.0.20220531/x86_64/mirror.list [Failed to connect to al2022-repos-us-east-2-9761ab97.s3.dualstack.us-east-2.amazonaws.com port 443 after 15200 ms: Connection timed out]
cloud-init[4858]: Ignoring repositories: amazonlinux
cloud-init[4858]: Package wget-1.21.2-2.amzn2022.x86_64 is already installed.
cloud-init[4858]: No match for argument: git
cloud-init[4858]: Package gcc-11.3.1-2.amzn2022.0.2.x86_64 is already installed.
cloud-init[4858]: Package make-1:4.3-5.amzn2022.x86_64 is already installed.
cloud-init[4858]: No match for argument: perf
cloud-init[4858]: No match for argument: python
cloud-init[4858]: Error: Unable to find a match: git perf python

Expected behavior All packages should be installed without error

[stewartsmith]

Are you sure you are running in a VPC that has connectivity to S3? Most commonly we see this kind of error when an instance is running in a VPC without an S3 endpoint configured that can point at the Amazon Linux repositories.

[talawahtech]

Are you sure you are running in a VPC that has connectivity to S3? Most commonly we see this kind of error when an instance is running in a VPC without an S3 endpoint configured that can point at the Amazon Linux repositories.

Yes, I am running in a public subnet (connected to an internet gateway) in the default VPC. Connectivity is fine once the instance boots up, and I am able to install those packages manually.

[talawahtech]

@stewartsmith reviewing the logs in greater details, it seems that dnf commands in the text/cloud-config section of the user data file fail, while those in the text/x-shellscript section work as expected.

[nmeyerhans]

Does the issue occur 100% of the time, or is it intermittent?

[talawahtech]

@nmeyerhans I am not sure. I had noticed the errors on previous infrastructure setup runs, but it happened in cases where the required packages were already installed, so it didn't cause a problem downstream, and I didn't pay super close attention.

My guess/intuition is that it is 100% of the time for dnf commands in the text/cloud-config section.

[talawahtech]

@nmeyerhans ok, I just got a chance to try it again and added dnf install -y wget git gcc make perf python to the top of the text/x-shellscript and it failed there with the same error as well.

Lower down in the same script another call to DNF works as expected. So doesn't seem to an issue of text/cloud-config vs text/x-shellscript, but one of timing.

[stewartsmith]

Do you still see anything like this with current AL2023 versions?