ApplicationInsights-JS icon indicating copy to clipboard operation
ApplicationInsights-JS copied to clipboard

URGENT ACTION: Stop using az416426.vo.msecnd.net

Open MSNev opened this issue 1 year ago • 8 comments

ACTION REQUIRED: Stop using az416426.vo.msecnd.net

To avoid any global OUTAGE you MUST change ALL of your CDN usage from “https://az416426.vo.msecnd.net/scripts/..” to our primary CDN endpoint https://js.monitor.azure.com/scripts/...

  • CDN Endpoint Change: All references to “az416426.vo.msecnd.net” must be updated to “js.monitor.azure.com” to prevent service disruptions due to a possible issue related to our current CDN configuration, this change MUST be done by Jan 15th, 2025.
  • We have identified some migration challenges with the legacy domain “az416426.vo.msecnd.net” which may lead to a temporary or permanent outages.
  • Action Required: Organizations must update their references and deploy changes to production before the service is disabled, and ensure that their sites can handle the absence of the Application Insights JS SDK from the CDN.

Right now we are expecting that there WILL be either a temporary or permanent outage of this domain based on the currently known details around the legacy domain.

What do you need to do?

  • Change ALL references of “az416426.vo.msecnd.net” to “js.monitor.azure.com”
  • Deploy the change to production before this external upcoming change occurs (we don’t control the timeframe)
  • Ensure that your site can handle when Application Insights JS Sdk cannot be loaded from the CDN without breaking your functionality

Details

Both the az416426.vo.msecnd.net and js.monitor.azure.com source their files from the same location as such all files and content is identical regardless of which domain endpoint you use.

Workarounds

  • Stop using “az416426.vo.msecnd.net” and change to “js.monitor.azure.com”
  • There is currently NO other workaround

We are currently investigating the available options on how we can avoid / migrate / mitigate this situation, but at this point it is HIGHLY likely that there will be either a temporary or permanent outage of this domain. As we currently have no known way to “migrate” this domain to a different CDN.

Any updates will be added to this issue.

MSNev avatar Dec 12 '24 20:12 MSNev

Why is this occurring?

Please see this public announcement Azure CDN from Edgio retirement FAQ where this domain (az416426.vo.msecnd.net is currently an Edgio based CDN instance) and because of the historical nature of this domain it's more complicated that most situations.

MSNev avatar Dec 20 '24 23:12 MSNev

Just to clarify - this affects only users who include the Application Insights JS SDK in their app by referencing the legacy CDN endpoint (e.g. using a script tag)?

baxxos avatar Jan 03 '25 14:01 baxxos

Correct, this issue ONLY affects users that are loading the SDK via the legacy CDN, if you are using the newer SDK Loader (snippet) which defaults to the current CDN endpoint (js.monitor.azure.com) or are using the SDK via an npm package embedded in your own bundles / domain then you are unaffected by this issue.

MSNev avatar Jan 06 '25 18:01 MSNev

Update: We have just started the process of redirecting this domain (hopefully) to avoid any possible outage, this change "should" take affect within the next couple of hours.

Known Issue (with Zero workaround -- ie. full outage)

  • The new CDN endpoint only support TLS 1.2+, so for any clients that do NOT support TLS 1.2 or greater the SSL connection will fail, we currently still support http on this endpoint.
  • Note: all of the other endpoints also only support TLS 1.2+ (which is why there is zero workaround (for TLS/SSL traffic)

Even if this is successful, this domain is now on an aggressive migration path and you should still continue to migrate away from this domain.

MSNev avatar Jan 06 '25 22:01 MSNev

The DNS (CNAME) entry has been redirected, and we appear to be seeing successful traffic being redirected to the new CDN.

However, the DNS Zone delegation still needs to be migrated, so there is still a risk of an outage...

MSNev avatar Jan 07 '25 01:01 MSNev

I've just "re-enabled" http access on this legacy endpoint (it had been defaulted to redirect all http -> https).

It appears that referrers that had been requesting http versions where successfully fetching the https version (as a follow up request from the same referrer)

MSNev avatar Jan 07 '25 01:01 MSNev

DNS Delegation has just been migrated and appears to be resolving as expected. One more final step scheduled for tomorrow (PST) to complete the migration process.

MSNev avatar Jan 15 '25 00:01 MSNev

The final mitigation step is now complete, significantly reducing the risk of an outage for this domain due to the Edgio retirement.

However, this domain is now entering a more orderly deprecation phase, with exact timeframes to be determined as we move beyond this unexpected situation.

MSNev avatar Jan 15 '25 17:01 MSNev