amazonlinux
[Bug] - yum/dnf update always resolving to s3.dualstack.amazonaws url
Describe the bug
I have a freshly deployed Amazon Linux 2023 instance that is placed in a VPC without direct internet access. The VPC has the s3 endpoint enabled with private DNS which should allow me to update this instance but whenever the yum update command is ran, it attempts to resolve a DNS record for a public facing IP address.
To Reproduce
- Connect to amazon linux 2023 EC2 instance
- Run
yum literallyanything
Expected behavior I should be able to use a private s3 endpoint without internet access to download updates from the mirror.
Screenshots
[root@ip-10-4-16-11 ~]# cat /etc/yum.repos.d/amazonlinux.repo
[amazonlinux]
name=Amazon Linux 2023 repository
mirrorlist=https://al2023-repos-us-east-1-de612dc2.s3.us-east-1.amazonaws.com/core/mirrors/$releasever/$basearch/mirror.list
priority=10
enabled=1
repo_gpgcheck=0
type=rpm
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-amazon-linux-2023
** snipped for brevity **
[root@ip-10-4-16-11 ~]# yum makecache
^CAmazon Linux 2023 repository [ === ] --- B/s | 0 B --:-- ET
Amazon Linux 2023 repository 0.0 B/s | 0 B 01:52
Errors during downloading metadata for repository 'amazonlinux':
- Curl error (28): Timeout was reached for https://al2023-repos-us-east-1-de612dc2.s3.dualstack.us-east-1.amazonaws.com/core/guids/bbefcd838a04819579bd2ca79d007905ca63d53b623c38b52e5fb51bf6aa91d6/x86_64/repodata/repomd.xml [Connection timeout after 30000 ms]
Error: Failed to download metadata for repo 'amazonlinux': Librepo was interrupted by a signal
Ignoring repositories: amazonlinux
Metadata cache created.
Desktop (please complete the following information):
- OS: amazon linux 2023
Additional context
Even when hardcoding the mirrorlist to use the ipv4 private vpc endpoint, when it retrieves the list it always attempts to connect out to the public IP addressed mirror. As you can see in the text box up there, even when hardcoding the URL to not include the .dualstack URL component, it's still being retrieved by something in the repolist
The reason that changing the mirrorlist entry doesn't help is that the content at the location it references is itself a URL, and it will also be relative to the dualstack endpoints.
In order to work around the behavior you're seeing, you'll need to hardcode the baseurl value instead of the mirrorlist. You can find the baseurl value with dnf repolist -v. Be aware that the GUID references a specific version of the repository, though, so you'll need to update it any time you want to point to a new AL2023 release.
What's actually happening in your case is that the VPC DNS server overrides the IPv4-only S3 endpoint DNS names, but does not override the corresponding dualstack name. I don't know why that is and we'll need to chase that down with the right team internally. I don't think there's anything we can do to make this work by default within the distro, since we explicitly want to default to using the dualstack endpoints for our repositories.
This is affecting me too, I'm also using an S3 VPC Endpoint. From the documentation, S3 VPC Endpoints don't support dualstack at this time, so it appears to me that this issue should cripple any Amazon Linux systems running yum from within a VPC with direct s3 access blocked.
@jpr-snl despite the documentation stating clearly that neither VPC endpoint type works, we had success accessing Amazon Linux 2023 repos over private connection using S3 Gateway Endpoint. Do you wanna give it a try?
On VPC Interface Endpoint it really doesn't work -- we have tried creating a Private Hosted Zone to trick AWS into sending dualstack access to our VPCe Interface, but the certs on the VPCe doesn't include dualstack...
After further investigation we realized that our NACL's were blocking some AWS S3 IP blocks. Fixing our NACL's resolved our issue.