AKS icon indicating copy to clipboard operation
AKS copied to clipboard

[Retirement] Migrate from acs-mirror.azureedge.net to packages.aks.azure.com - Egress requirement change for AKS clusters

Open rahulrai-in opened this issue 9 months ago • 9 comments

Due to the Edgio retirement, Azure Kubernetes Service (AKS) is updating its egress requirements. The FQDN acs-mirror.azureedge.net:443—currently used to download and install binaries for necessary dependencies (e.g., kubenet and Azure CNI)—will experience degraded performance starting April 30, 2025, and will be fully retired on May 31, 2025.

Required action

If you use a Layer 7 (FQDN) firewall to restrict AKS egress traffic, action may be required:

1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

2. Other Firewalls or Egress Restrictions If you restrict egress using another solution (not Azure Firewall as described above), you must add packages.aks.azure.com:443 to your allowed egress list by May 31, 2025. After this date, new AKS VHD images will require access to this hostname to successfully provision.

Updated Egress rules for AKS clusters are available in the Microsoft Learn documentation.

FAQs

Q. What are the destination IP addresses of the host packages.aks.azure.com and acs-mirror.azureedge.net?

Because of the CDN’s dynamic nature, these fully qualified domain names (FQDNs) can map to different Point of Presence (PoP) IP addresses over time or use multiple IPs simultaneously for optimal performance and high availability. As a result, AKS does not guarantee a fixed or static set of destination IP addresses. We recommend that customers allow outbound access to the FQDNs packages.aks.azure.com and acs-mirror.azureedge.net through a Layer 7 network appliance such as the Azure Firewall.

Q. What happens if I do not update my firewall to allow access to the new FQDN by May 31, 2025?

When you perform cluster or Nodepool upgrades, your AKS cluster nodes will use the new Virtual Hard Disk (VHD) image and download necessary binaries from the new FQDN. If you do not update your firewall to allow access to the new FQDN by the full deprecation date, May 31st, 2025, the nodes will fail to be provisioned, and your upgrade operation will fail. However, the existing nodes will continue to function until they are replaced by AKS Node Auto-Healing or a Nodepool upgrade.

Q. How long do I need to allow access to the old FQDN acs-mirror.azureedge.net:443?

New VHD images that reference the new FQDN-packages.aks.azure.com will be released in March 2025. After the new VHD images are applied to all the nodes of your AKS cluster, you can safely remove access to the old FQDN.

Q. What happens if my AKS cluster has the node auto-upgrade OS feature enabled, and it updates the cluster nodes to the new VHD that accesses the new FQDN before I add the new FQDN to my firewall rules?

We are implementing a fallback logic in the new VHDs that tries to access the old FQDN if it fails to access the new FQDN-packages.aks.azure.com. You might experience a small latency due to this fallback operation, but your node will still successfully provision.

Q. What happens if my cluster cannot upgrade to the new VHD image that accesses the new FQDN?

Please note that after May 31,2025, the acs-mirror.azureedge.net CDN endpoint will no longer function and therefore any operation that that causes the nodes to be replaced-such as the Node Auto-Healing-will fail to operate. If you are on a legacy Kubernetes cluster, we recommend following our extensive AKS Migration Guide to migrate to a new AKS cluster.

Help and Support

If you have questions, get answers from community experts in Microsoft Q&A. If you have a support plan and you need technical help, create a support request.

rahulrai-in avatar Feb 12 '25 02:02 rahulrai-in

Any changes to these IP addresses will be communicated at least one week in advance.

How will those changes be communicated?

ms1111 avatar Feb 22 '25 21:02 ms1111

Any changes to these IP addresses will be communicated at least one week in advance.

How will those changes be communicated?

Thanks for your comment @ms1111. Information regarding any changes to the IP addresses will be available in AKS release notes.

rahulrai-in avatar Feb 24 '25 04:02 rahulrai-in

  1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

TomkhaLoL avatar Feb 24 '25 08:02 TomkhaLoL

  1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

I think this service tag is referring to "after change", not "before change".

If you check the IP result from two different domains, the deprecated one is not in the list.

JoeyC-Dev avatar Feb 24 '25 12:02 JoeyC-Dev

  1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

Since today we're getting alerts that requests from our k8s cluster to packages.aks.azure.com are getting blocked, but we do have the FQDN Tag "AzureKubernetesServices" whitelisted in our Azure Firewall. Is this update of the FQDN tag still pending or do we maybe have to act here?

I've been seeing that since last week and am trying to persuade Azure support that there is an issue!

rlaveycal avatar Feb 24 '25 16:02 rlaveycal

1. Azure Firewall with AzureKubernetesService FQDN Tag If you rely on Azure Firewall Application rules with the AzureKubernetesService FQDN tag, no action is needed. The update will be applied automatically.

@rahulrai-in the ServiceTag is currently not working, is this pending, or should it be completed already?

I've raised a support ticket with Azure about this:

Support request ID	2502240050004619
Created on	Mon, Feb 24, 2025, 5:32:47 PM
Support plan	Unified Enterprise

toredash avatar Feb 24 '25 21:02 toredash

@toredash @rlaveycal @JoeyC-Dev @TomkhaLoL Thanks for the feedback. The FQDN tag updates are yet to be deployed. Once deployed the change will be automatically applied to your Firewall if your Firewall rules allow access to the AzureKubernetesService tag.

rahulrai-in avatar Feb 24 '25 22:02 rahulrai-in

@rahulrai-in What is the timeline for the FQDN tag update ?

cfrost avatar Feb 26 '25 14:02 cfrost

@rahulrai-in What is the timeline for the FQDN tag update ?

Currently we are expecting to complete the roll out of the tag update in March'25.

rahulrai-in avatar Feb 27 '25 23:02 rahulrai-in

Will it be enough to use Service tag "AzureCloud" in NSG to allow access to packages.aks.azure.com? Because now this is the tag we use, and it is working with acs-mirror.azureedge.net

PaulBrand92 avatar May 11 '25 08:05 PaulBrand92

Will it be enough to use Service tag "AzureCloud" in NSG to allow access to packages.aks.azure.com? Because now this is the tag we use, and it is working with acs-mirror.azureedge.net

NSG tags are different from Firewall tags. The AzureKubernetesService tag is applicable to Firewall only.

rahulrai-in avatar Jun 04 '25 07:06 rahulrai-in

This announcement has been automatically marked as stale because it has not had any activity for 90 days. It will be closed if no further activity occurs within 7 days of this comment.