core icon indicating copy to clipboard operation
core copied to clipboard

Critical: .NET install domains and URLs are changing

Open richlander opened this issue 11 months ago • 51 comments

.NET binaries and installers have moved to a new domain -- builds.dotnet.microsoft.com -- backed by a new Content Delivery Network (CDN). You may need to make changes to adjust.

Tracking issue: https://github.com/dotnet/core/issues/9674

.NET Team Status:

  • azureedge.net scream tests are starting
  • See: https://github.com/dotnet/core/issues/9676

Recommended action:

Last update: 2025.01.23 Next update: 2025.02.04

Context

Some .NET binaries and installers have been hosted on Azure Content Delivery Network (CDN) domains that end in .azureedge.net. These domains were hosted by edg.io, which has ceased operations due to bankruptcy. We migrated to a new CDN and will be using new domains going forward.

We expectazureedge.net domains to cease by 5/31 (at latest).

.NET Team Remediation

New domains were created:

  • builds.dotnet.microsoft.com for signed official builds
  • ci.dot.net for unsigned builds

Test links for new CDN:

  • https://builds.dotnet.microsoft.com/dotnet/health.txt
  • https://builds.dotnet.microsoft.com/dotnet/release-metadata/releases-index.json

Test links for old CDN:

  • https://dotnetcli.azureedge.net/dotnet/health.txt
  • https://dotnetcli.azureedge.net/dotnet/release-metadata/releases-index.json

The following resources have been updated to use new domains.

Other changes are in progress.

User Remediation

Please make the following replacements:

  • dotnetcli.azureedge.net -> builds.dotnet.microsoft.com
  • dotnetcli.blob.core.windows.net -> builds.dotnet.microsoft.com
  • dotnetbuilds.azureedge.net -> ci.dot.net
  • dotnetbuilds.blob.core.windows.net -> ci.dot.net

All the new domains are path-compatible with old domains, as they share the same origin.

Please also make the following changes:

  • Update local copies of the install script
  • Update setup-dotnet Action references
  • Validate that firewall rules work with the new domains

Install script

The .NET install script is used to install .NET from our CDN. We are changing CDNs (documented in a following section), which requires us to change the install script to use the new CDN.

Updated scripts:

  • https://dot.net/v1/dotnet-install.sh
  • https://dot.net/v1/dotnet-install.ps1

Notes:

  • Users who have local copies of these scripts will need to update their copies.
  • Users who rely on the remote copy (at the URLs above) do not need to do anything other than validate no observed change in behavior (due to new domains and CDNs being used).

Tracking PRs:

Official builds

Official builds and JSON files are hosted via a CDN, available for use by the install script and other installers.

  • New: https://builds.dotnet.microsoft.com
  • Old: https://dotnetcli.azureedge.net

Notes:

You can change from old to new domains by changing the domain section of the URL. The other parts of the URL do not need to change.

Example URLs:

  • New: https://builds.dotnet.microsoft.com/dotnet/Runtime/8.0.11/dotnet-runtime-8.0.11-linux-x64.tar.gz
  • Old: https://dotnetcli.azureedge.net/dotnet/Runtime/8.0.11/dotnet-runtime-8.0.11-linux-x64.tar.gz
  • New: https://builds.dotnet.microsoft.com/dotnet/release-metadata/releases-index.json
  • Old: https://dotnetcli.azureedge.net/dotnet/release-metadata/releases-index.json

A set of short links are available for official builds.

Link pattern:https://aka.ms/dotnet/[x.y]/[package].

Example URLs:

  • https://aka.ms/dotnet/9.0/dotnet-sdk-win-x64.zip
  • https://aka.ms/dotnet/8.0/aspnetcore-runtime-linux-x64.tar.gz

These links produce301 HTTP results that forward to our CDN.

  • Currently forward to: https://dotnetcli.azureedge.net.
  • Will be changed to forward to: https://build.dotnet.microsoft.com.

We expect these links to be changed in mid January.

Tracking PR:

CI builds

Continuous integration (CI) builds are hosted via a CDN, available via the install script and GitHub README files.

  • New: https://ci.dot.net
  • Old: https://dotnetbuilds.azureedge.net

Note: CI builds include a mix of tested and untested builds, signed and unsigned builds.

Example URLs:

  • New: https://ci.dot.net/public/Runtime/9.0.1-servicing.24610.11/dotnet-runtime-9.0.1-win-x64.exe
  • Old: https://dotnetbuilds.azureedge.net/public/Runtime/9.0.1-servicing.24610.11/dotnet-runtime-9.0.1-win-x64.exe

A set of short links are available for CI builds.

Link pattern:https://aka.ms/dotnet/[x.y]/daily/[package].

Example URLs:

  • https://aka.ms/dotnet/9.0/daily/dotnet-runtime-win-x64.exe
  • https://aka.ms/dotnet/9.0/daily/aspnetcore-runtime-linux-arm.tar.gz
  • https://aka.ms/dotnet/10.0.1xx/daily/dotnet-sdk-osx-arm64.pkg

These links produce301 HTTP results that forward to our CDN.

  • New builds now forward to: https://ci.dot.net
  • Old builds forward to: https://dotnetbuilds.azureedge.net.

We expect these links to be changed in mid January.

Tracking PR:

CI build pages use the CI short links.

Example build pages:

Azure DevOps and GitHub Actions

  • Major versions tags for actions/setup-dotnet have been updated. References to pinned versions will require updating to the most recent version.
  • We expect that GitHub Enterprise Server will be addressed in January.
  • Azure DevOps UseDotnetTask will be updated in January
  • We do not yet have a date for updating Azure DevOps Server.

richlander avatar Dec 24 '24 02:12 richlander

Is there a risk that a malicious party later acquires azureedge.net and starts serving malware to systems that still use the old URLs? From WHOIS, it looks like azureedge.net is registered to Microsoft, not to Edgio. (Just wondering how urgent it is to update URLs in old version-control branches that are not actively developed but might get built some day.)

Have there been any NuGet feeds in the domain?

KalleOlaviNiemitalo avatar Dec 24 '24 07:12 KalleOlaviNiemitalo

@KalleOlaviNiemitalo we took it over, so it won’t be taken away.

shanselman avatar Dec 24 '24 08:12 shanselman

@KalleOlaviNiemitalo we took it over, so it won’t be taken away.

so why not keep the current urls for like 1-2 more dont net versions so after .net 11 you have to use the new urls this would give people time to update their whitelists

zarlo avatar Dec 24 '24 08:12 zarlo

Given this issue I'm wondering when Microsoft will provide their own domain registrar on Azure to prevent such issues in the future. Currently this is the only thing that is really missing on the Azure platform. I can host virtually anything on Azure but when it comes to domains I still need to resort to a third party. I can point all my nameservers to Azure, sure. But the domain itself needs to be hosted somewhere else.

klemmchr avatar Dec 24 '24 09:12 klemmchr

Does this affect the installers in the Azure Devops pipelines? We use a mix of classic and Yaml pipelines.

charles-Graham-Keilman avatar Dec 24 '24 14:12 charles-Graham-Keilman

Does this affect the installers in the Azure Devops pipelines? We use a mix of classic and Yaml pipelines.

Yes, it does.

Azure DevOps and GitHub Actions installation tools are dependent on some of these resources. We are working directly with those teams to maintain continuity of service. They are moving to the new domains at best speed.

klemmchr avatar Dec 24 '24 15:12 klemmchr

Users should not consider azureedge.net to be a long-term usable domain. Please move to the new domains as soon as possible. It is likely that these domains will be retired, however, no other party will be able to use them. We are not able to control the timing of these events.

I updated the issue with that text.

Does this affect the installers in the Azure Devops pipelines?

We are working closely with the Azure DevOps and GitHub Actions teams. They plan to start deploying changes to use the new domains on the 26th. You can see their PRs backreferenced in this issue, above. Note that this situation is fluid, so that date could change. Both Azure DevOps and GitHub Actions have on-prem scenarios. We'll start addressing that in the new year.

Have there been any NuGet feeds in the domain?

You can actually answer some of these questions with nslookup or dig.

$ nslookup api.nuget.org
Server:		100.100.100.100
Address:	100.100.100.100#53

Non-authoritative answer:
api.nuget.org	canonical name = nugetapiprod.trafficmanager.net.
nugetapiprod.trafficmanager.net	canonical name = apiprod-mscdn.azureedge.net.
apiprod-mscdn.azureedge.net	canonical name = apiprod-mscdn.afd.azureedge.net.
apiprod-mscdn.afd.azureedge.net	canonical name = azureedge-t-prod.trafficmanager.net.
azureedge-t-prod.trafficmanager.net	canonical name = shed.dual-low.s-part-0042.t-0009.t-msedge.net.
shed.dual-low.s-part-0042.t-0009.t-msedge.net	canonical name = azurefd-t-fb-prod.trafficmanager.net.
azurefd-t-fb-prod.trafficmanager.net	canonical name = dual.s-part-0042.t-0009.fb-t-msedge.net.
dual.s-part-0042.t-0009.fb-t-msedge.net	canonical name = s-part-0042.t-0009.fb-t-msedge.net.
Name:	s-part-0042.t-0009.fb-t-msedge.net
Address: 13.107.253.70

You can see that NuGet has a dependency as well, but they are in a much better position since it is an implementation detail that they can control transparently. Everything is behind a custom domain, which is the correct architecture. I suspect that you will soon see edgio disappear and Akamai appear in that list.

You can see that our new domain is all on Azure Front Door.

$ nslookup builds.dotnet.microsoft.com
Server:		100.100.100.100
Address:	100.100.100.100#53

Non-authoritative answer:
builds.dotnet.microsoft.com	canonical name = dotnetcli.trafficmanager.net.
dotnetcli.trafficmanager.net	canonical name = dotnetcli-f0e9dzh5e5eze7cd.b02.azurefd.net.
dotnetcli-f0e9dzh5e5eze7cd.b02.azurefd.net	canonical name = shed.dual-low.s-part-0042.t-0009.t-msedge.net.
shed.dual-low.s-part-0042.t-0009.t-msedge.net	canonical name = azurefd-t-fb-prod.trafficmanager.net.
azurefd-t-fb-prod.trafficmanager.net	canonical name = dual.s-part-0042.t-0009.fb-t-msedge.net.
dual.s-part-0042.t-0009.fb-t-msedge.net	canonical name = s-part-0042.t-0009.fb-t-msedge.net.
Name:	s-part-0042.t-0009.fb-t-msedge.net
Address: 13.107.253.70

Our existing domain is pointed at an Azure Traffic Manager and then to edgio. If edgio goes down, we can point the traffic manager at AFD or Akamai (once we have service there).

$ nslookup dotnetcli.azureedge.net    
Server:		100.100.100.100
Address:	100.100.100.100#53

Non-authoritative answer:
dotnetcli.azureedge.net	canonical name = dotnetcliedge.trafficmanager.net.
dotnetcliedge.trafficmanager.net	canonical name = cs9.wpc.v0cdn.net.
Name:	cs9.wpc.v0cdn.net
Address: 72.21.81.200

I hope the extra details help.

richlander avatar Dec 24 '24 16:12 richlander

When I asked about NuGet, I meant non-default feed URLs that might have been added to NuGet.Config files and would have to be updated. For example, https://github.com/dotnet/command-line-api/blob/main/README.md#daily-builds suggests adding https://pkgs.dev.azure.com/dnceng/public/_packaging/dotnet-libraries/nuget/v3/index.json to nuget.config; this URL does not refer to the azureedge.net domain, but perhaps some other project advertises an azureedge.net URL for its preview NuGet feed.

KalleOlaviNiemitalo avatar Dec 25 '24 11:12 KalleOlaviNiemitalo

Great question and thanks for the additional detail. We haven't heard of any such breakage example. It's certainly possible.

There are many other teams (and companies) with this same problem. The way that their impacted scenarios surface will be quite varied.

richlander avatar Dec 25 '24 17:12 richlander

FYI @dotnet/distro-maintainers

richlander avatar Dec 26 '24 06:12 richlander

Azure DevOps UseDotnetTask will be updated in January We also noticed that there is a lot of use of our storage account: dotnetcli.blob.core.windows.net. Please also search for it. The storage account is unaffected, however, it would be much better for everyone if you used our new CDN. It will deliver better peformance.

@richlander I noticed your pipeline team only updated the download links. Did they maybe miss something or not fully understand the issue? Could you also fix the release index links when you push out the update in January?

Varorbc avatar Dec 27 '24 04:12 Varorbc

I will share this with them @Varorbc.

richlander avatar Dec 27 '24 04:12 richlander

maybe a noob question why didn’t just keep the old domain? why would a domain change be needed? couldn’t the old domain name not simply resolved to the new servers?

Rand-Random avatar Dec 27 '24 07:12 Rand-Random

We asked the same question. We were told that this option wasn't being made available. We don't have more information on that.

richlander avatar Dec 27 '24 10:12 richlander

What's the difference between getting official builds from builds.dotnet.microsoft.com and download.visualstudio.microsoft.com? Most of the links in the JSON files seem to point to the latter.

chrarnoldus avatar Dec 27 '24 13:12 chrarnoldus

No difference. Both are fine. download.visualstudio.microsoft.com is/was unaffected by this situation.

We'll be publishing new guidance after we've had a chance for some "downtime". It's likely that the new guidance will apply more to how the install script, GitHub Actions, and AzDo Tasks are implemented than requiring a typical user to do something significantly different.

richlander avatar Dec 27 '24 15:12 richlander

It is unfortunate but understandable that MS is now in full control of the azureedge.net domain yet is unable to setup redirects. A bit more confusing is this part:

URLs affected:

  • https://dot.net/v1/dotnet-install.sh
  • https://dot.net/v1/dotnet-install.ps1
  • https://aka.ms/dotnet/ (various)

These are already aliases, aka.ms is literally a redirect service. Can these urls not be updated to the correct locations? Infact it looks like some already are redirect to "not affected" domains:

wget2 https://dot.net/v1/dotnet-install.ps1
HTTP response 301  [https://dot.net/v1/dotnet-install.ps1]
URL 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.ps1'

wget2 https://dot.net/v1/dotnet-install.sh
HTTP response 301  [https://dot.net/v1/dotnet-install.sh]
URL 'https://dotnet.microsoft.com/download/dotnet/scripts/v1/dotnet-install.sh'

Should they be removed from the affected list?

It does look like several of the aka.ms urls will 301 redirect as mentioned in this ticket already or showing in the actual location 301 returned that is is pointing to a safe domain.

mitchcapper avatar Dec 27 '24 17:12 mitchcapper

@mitchcapper the contents of those scripts were changed to use the new CDNs. It should be transparent to most folks, but depending on how your infrastructure is set up like allow lists, copy of the scripts, etc. you might need to take action.

mairaw avatar Dec 27 '24 17:12 mairaw

I updated the content above. It addresses the change in the install script. Thanks for asking for that clarification @mitchcapper. Good question.

richlander avatar Dec 27 '24 17:12 richlander

Something maybe worth emphasizing is that consumers need to make sure they aren't using an overly-specific version of the action in order to receive this update.

In our community we found many instances of people explicitly using v4.0.0 instead of v4 due to a commonly copy+pasted workflow. Similar mistakes appear in several repos in the dotnet org and on GitHub in general.

PathogenDavid avatar Jan 03 '25 16:01 PathogenDavid

Good point. I made a couple edits to clarify.

richlander avatar Jan 03 '25 16:01 richlander

Are you making any changes to docker images?

PS > docker pull mcr.microsoft.com/dotnet/sdk:8.0
8.0: Pulling from dotnet/sdk
fd674058ff8f: Already exists
bbdcc09b9436: Already exists
22a0038f868f: Already exists
4416e5d0ee0a: Already exists
856be9b83b46: Already exists
61ccb076d3e2: Already exists
879914dae944: Downloading  22.15MB
bdc2dbb58306: Retrying in 1 second
dd4aaba891e2: Downloading  10.26MB
error pulling image configuration: download failed after attempts=6: EOF
PS > docker pull mcr.microsoft.com/dotnet/sdk:8.0
Error response from daemon: Get "https://mcr.microsoft.com/v2/dotnet/sdk/manifests/sha256:f25e4f51fa06e3b14af1a1135013a3e96055b76caa0e76afc0096d64a77879fd": EOF
PS > docker pull mcr.microsoft.com/dotnet/sdk:8.0
Error response from daemon: Head "https://mcr.microsoft.com/v2/dotnet/sdk/manifests/8.0": EOF

voroninp avatar Jan 03 '25 22:01 voroninp

This is the outstanding PR for our container images: https://github.com/dotnet/dotnet-docker/pull/6125.

However, your error looks like a standard network problem. It has nothing to do with this issue. None of the problematic changes have been made yet and I don't believe that MCR uses edgio CDN or the azureedge.net domain.

richlander avatar Jan 03 '25 23:01 richlander

The command npx playwright install redirects to https://playwright.azureedge.net/*. Is the install affected by https://devblogs.microsoft.com/dotnet/critical-dotnet-install-links-are-changing and necessary to update the links?

Odraio avatar Jan 08 '25 19:01 Odraio

@Odraio I believe these urls were updated in https://github.com/microsoft/playwright/pull/34061, perhaps if you ask on their repo they will have information about release cadences?

baronfel avatar Jan 08 '25 19:01 baronfel

@richlander For our Azure Functions, we currently use the URL functionscdn.azureedge.net to retrieve extension bundles. However, after reviewing the issue, I noticed that there is no specific mention of the subdomain functionscdn in the migration details. Could you please clarify if this subdomain will also be migrated and, if so, to which domain?

For example, we are getting extension bundles from a url like: "https://functionscdn.azureedge.net/public/ExtensionBundles/Microsoft.Azure.Functions.ExtensionBundle.Preview/$EXTENSION_BUNDLE_VERSION"

cetinbug avatar Jan 09 '25 14:01 cetinbug

We use the URL https://azcopyvnext.azureedge.net for downloading Azcopy, as it mentioned in all of MS's documentation as the location for downloading from. This stopped working yesterday and is not mentioned in your update. What is the new URL for this?

asatrur avatar Jan 10 '25 14:01 asatrur

@cetinbug: Maybe it's better to ask in one of these repos:


@asatrur: https://github.com/Azure/azure-storage-azcopy/issues/2903.

The aka ms links will now point to URLs that start with https://azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net/

If using the aka.ms links, they already point to azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net.

o-l-a-v avatar Jan 10 '25 14:01 o-l-a-v

@asatrur: Azure/azure-storage-azcopy#2903.

The aka ms links will now point to URLs that start with https://azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net/

If using the aka.ms links, they already point to azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net.

* https://www.whatsmydns.net/redirect-checker?q=https%3A%2F%2Faka.ms%2Fdownloadazcopy-v10-windows

Thanks, it would have been nice if they mentioned that somewhere before screwing us

asatrur avatar Jan 10 '25 14:01 asatrur

@asatrur: Azure/azure-storage-azcopy#2903.

The aka ms links will now point to URLs that start with https://azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net/

If using the aka.ms links, they already point to azcopyvnext-awgzd8g7aagqhzhe.b02.azurefd.net.

* https://www.whatsmydns.net/redirect-checker?q=https%3A%2F%2Faka.ms%2Fdownloadazcopy-v10-windows

Thanks, it would have been nice if they mentioned that somewhere before screwing us

Please don't pull from any of the *.azureedge.net or *.azurefd.net for any service - you should be using the custom domains that front the CDN endpoints or the canonical aka.ms links. In the case of AzCopy, those links can always be found at https://aka.ms/downloadazcopy. Each platform we distribute for has its own aka.ms link that points to the most current release.

schoag-msft avatar Jan 10 '25 22:01 schoag-msft