winget-cli icon indicating copy to clipboard operation
winget-cli copied to clipboard

Publishing is down again (failed to update source)

Open infinitepower18 opened this issue 2 years ago • 33 comments

Brief description of your issue

Same problem that previously occured #2666 has started again. And downloading the msix from https://cdn.winget.microsoft.com/cache/source.msix returns a 0kb file.

Steps to reproduce

Run winget source update

Expected behavior

winget should have updated successfully

Actual behavior

Winget will attempt to update the source when running any command but will return the message Failed in attempting to update the source: winget. If you run winget source update it will respond with Cancelled

Environment

Windows Package Manager v1.3.2691
Windows: Windows.Desktop v10.0.22621.819
System Architecture: X64
Package: Microsoft.DesktopAppInstaller v1.18.2691.0

infinitepower18 avatar Nov 22 '22 22:11 infinitepower18

FYI: https://github.com/microsoft/winget-pkgs/issues/86160#issuecomment-1323964593

mdanish-kh avatar Nov 22 '22 22:11 mdanish-kh

+1

Vzz1c avatar Nov 23 '22 09:11 Vzz1c

+1

Psi085 avatar Nov 23 '22 14:11 Psi085

+1

simkoG avatar Nov 23 '22 16:11 simkoG

winget source list
-----------------------------------------------------
msstore https://storeedgefd.dsx.mp.microsoft.com/v9.0
winget  https://cdn.winget.microsoft.com/cache

ResourceNotFoundThe specified resource does not exist.

M43R avatar Nov 24 '22 12:11 M43R

I have the same problem "Failed in attempting to update the source: winget". After following this solution, Winget will work perfectly.

vtvy avatar Nov 24 '22 17:11 vtvy

Another workaround is to use a VPN. Use a location far from your current continent.

mgthantzin avatar Nov 25 '22 06:11 mgthantzin

My bad, that was my UK VPN previously, sorry. It is working if I connect to a VPN server far away from me.

simkoG avatar Nov 25 '22 07:11 simkoG

When will this be fixed?

VincentJoshuaET avatar Nov 26 '22 07:11 VincentJoshuaET

We believe we've found the root cause. We've triggered another purge to reset the cache endpoints and we're working with our CDN provider to ensure all regions are updated. We've got a production change schedule for Monday to hopefully address this permanently.

denelon avatar Nov 26 '22 19:11 denelon

We've gotten an update on the CDN endpoints. There were two endpoints that needed to be manually purged. We've been told these have been completed.

denelon avatar Nov 28 '22 19:11 denelon

Still not working for me.

PS C:\Users\User\code\BoxSetup> winget source reset --force
Resetting all sources...Done
PS C:\Users\User\code\BoxSetup> winget source update
Updating all sources...
Updating source: msstore...
Done
Updating source: winget...
Cancelled

Downloading https://cdn.winget.microsoft.com/cache/source.msix through a web browser produces a 0KB file.

MarcoPeraza avatar Nov 28 '22 22:11 MarcoPeraza

Working now

MarcoPeraza avatar Nov 29 '22 17:11 MarcoPeraza

All, we've had our CDN provider flush a couple of the pesky nodes that weren't getting updated, and we're purging after each publish run. This should have been resolved around 5:00 PM Pacific yesterday.

denelon avatar Nov 29 '22 17:11 denelon

@denelon It is fixed for me (Hungary, EU).

simkoG avatar Nov 29 '22 17:11 simkoG

Still not working for me. Getting 0 byte file from the CDN in US.

cdn.winget.microsoft.com resolves to 152.195.19.97 for me.

efie45 avatar Dec 19 '22 16:12 efie45

i posted the issue #2807 referenced above and it was pointed out to me that there is already a pinned issue ... so i experimented a bit more with the info i found in here.

(GitHub places the pinned issues OUTSIDE the list of current issues - i did not even notice it because it is placed in such a way that is confused with an advertising banner... and i developed a habit to ignore anything placed in such locations... thus the pinned issues area is usually ignored)

i tried downloading the https://cdn.winget.microsoft.com/cache/source.msix file with:

  • internet explorer: result = error INET_E_DOWNLOAD_FAILURE
  • MS Edge Chromium, result = downloads ok
  • Google Chrome, result = downloads ok
  • Mozilla Firefox, result = downloads ok
  • winget upgrade --verbose-logs command line (winget v1.4.3531), result = Failed in attempting to update the source: winget An error occurred in the secure channel support
2022-12-28 21:26:41.640 [REPO] Source past auto update time [5 mins]; it has been at least 27870926 mins
2022-12-28 21:26:42.095 [FAIL] WindowsPackageManager.dll!00007FF89ECBBF22: ReturnHr(1) tid(3c90) 80072F7D     Msg:[winrt::hresult_error: An error occurred in the secure channel support] 

2022-12-28 21:26:42.096 [FAIL] WindowsPackageManager.dll!00007FF89EB43CF1: LogHr(2) tid(3c90) 80072F7D 
2022-12-28 21:26:42.096 [FAIL] D:\a\_work\1\s\external\pkg\src\AppInstallerRepositoryCore\RepositorySource.cpp(53)\WindowsPackageManager.dll!00007FF89ECC7451: (caller: 00007FF89EB906FE) LogHr(3) tid(3c90) 80072F7D     Msg:[winrt::hresult_error: An error occurred in the secure channel support] 

2022-12-28 21:26:42.096 [REPO] Source add/update failed, waiting a bit and retrying: winget
2022-12-28 21:29:45.049 [FAIL] WindowsPackageManager.dll!00007FF89ECBBF22: ReturnHr(2) tid(3c90) 80072F7D     Msg:[winrt::hresult_error: An error occurred in the secure channel support] 

2022-12-28 21:29:45.049 [FAIL] WindowsPackageManager.dll!00007FF89EB43CF1: LogHr(5) tid(3c90) 80072F7D 
2022-12-28 21:29:45.050 [FAIL] D:\a\_work\1\s\external\pkg\src\AppInstallerRepositoryCore\RepositorySource.cpp(540)\WindowsPackageManager.dll!00007FF89ECC6A33: (caller: 00007FF89EA88B3D) LogHr(6) tid(3c90) 80072F7D     Msg:[winrt::hresult_error: An error occurred in the secure channel support] 

2022-12-28 21:29:45.050 [REPO] Failed to update source: winget

ping cdn.winget.microsoft.com

Pinging sni1gl.wpc.nucdn.net [152.199.21.175] with 32 bytes of data:
Reply from 152.199.21.175: bytes=32 time=6ms TTL=58
Reply from 152.199.21.175: bytes=32 time=6ms TTL=58
Reply from 152.199.21.175: bytes=32 time=6ms TTL=58
Reply from 152.199.21.175: bytes=32 time=5ms TTL=58

Ping statistics for 152.199.21.175:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 5ms, Maximum = 6ms, Average = 5ms

image

R-Adrian avatar Dec 28 '22 19:12 R-Adrian

@R-Adrian thanks! That's some helpful information. I wonder if something with the user agent might be at play. I'll share this with the engineering team.

denelon avatar Dec 28 '22 20:12 denelon

i think i nailed it... the error message is almost correct.... it is a SChannel problem but is actually a problem with Windows (10? 22H2?) ... the operating system is not actually TLS 1.3 compliant and has to use TLS 1.2 The download via the modern browsers works because the browser handles TLS 1.3 internally... but internet explorer and winget rely on the OS to establish the TLS connection... which is not working.

I managed to reproduce the issue on another computer like this:

a) defined the registry key

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\SecurityProviders\SCHANNEL\Protocols\TLS 1.3\Client]
"DisabledByDefault"=dword:00000000
"Enabled"=dword:00000001

b) rebooted the system

c) and then managed to cause the schannel winget error by simply enabling the TLS 1.3 checkbox in the Advanced Internet Options ( inetcpl.cpl ) while at the same time having the TLS 1.3 SChannel registry setting above.

image

d) when the TLS 1.3 checkbox is disabled in inetcpl.cpl, even if the SChannel registry settings for TLS 1.3 client remain enabled the checkbox overrides the registry setting, and winget (and internet explorer) works again... but it uses TLS 1.2 connections

image

R-Adrian avatar Dec 28 '22 21:12 R-Adrian

P.S. found further support material, TLS 1.3 seems is not intended to be supported in Windows 10

https://learn.microsoft.com/en-us/windows/win32/secauthn/protocols-in-tls-ssl--schannel-ssp-

image

R-Adrian avatar Dec 28 '22 22:12 R-Adrian

P.S. 2.. after sleeping on it.. i think this TLS 1.3 bug with Microsoft Azure connections might also be the reason for Windows Update and the Microsoft Store app itself randomly failing instantly when you open them during the past few months ... and then they start to mysteriously work again if you leave them alone for a few minutes without doing anything to them. (i already checked NTP clock sync, that is not the issue)

On some computers at the office i configured them to use a proxy for updates and my squid proxy logs are full of failed tunnel connections to these endpoints even if i have allowed them in the proxy config and they work via the modern browsers: slscr.update.microsoft.com:443 sls.update.microsoft.com:443 not surprisingly, i have also found the cdn.winget.microsoft.com:443 endpoint in the squid proxy logs there.

R-Adrian avatar Dec 29 '22 06:12 R-Adrian

I get the same error on Windows 11 Home 22H2.

C:\Users\DaveM>winget upgrade --verbose-logs
Failed in attempting to update the source: winget

cdn.winget.microsoft.com resolves to 152.199.21.175 for me. winget version v1.3.2691

BuzzwordChief avatar Dec 30 '22 15:12 BuzzwordChief

I get the same error on Windows 11 Home 22H2.

can you please try temporarily disabling TLS 1.3 from the Internet Options -> Advanced control panel and trying again with that? (start->run-> inetcpl.cpl) Make sure TLS 1.2 remains enabled there.

(i do not have any Windows 11 system to test with)

R-Adrian avatar Dec 30 '22 15:12 R-Adrian

Does not seem to help: image

Do I have to restart or anything?

BuzzwordChief avatar Dec 30 '22 15:12 BuzzwordChief

Do I have to restart or anything?

Not sure how Win11 behaves here, i only had to restart when i enabled TLS 1.3 in registry for SChannel, not when using the control panel on Win10 22H2.

Maybe try completely disabling TLS 1.3 via the registry options for SChannel and then rebooting?

R-Adrian avatar Dec 30 '22 15:12 R-Adrian

Happy New Year, and i tried testing another thing...

  • on Win 10, toggling TLS 1.3 on/off in inetcpl.cpl still causes winget errors when TLS 1.3 is on
  • according to Qualys SSL Labs https://cdn.winget.microsoft.com appears to be serving different content depending on the connection being on TLS 1.2 or TLS 1.3 and also if SNI is negotiated or not negotiated. https://www.ssllabs.com/ssltest/analyze.html?d=cdn.winget.microsoft.com

i cannot make SSLLabs analyze the European side of the CDN - SSLLabs does not let me select 152.199.21.175 and it auto-selects to the North America one, 152.195.19.97.. so i added that to my hosts file and tested again with that.

result: Same error happens with winget and 152.195.19.97 present in the hosts file (on Windows 10 22H2)

image

image

image

R-Adrian avatar Jan 01 '23 09:01 R-Adrian

+1 Solutions didn't work for me.

fluentmoheshwar avatar Jan 12 '23 17:01 fluentmoheshwar

We've been working with our CDN provider to squash the nasty zero-byte file being cached and served. Is anyone still getting this problem? If you are, can you share the IP address you're hitting?

denelon avatar Jan 26 '23 21:01 denelon

TLS 1.3 error is still there - the first few tries are with TLS 1.3 enabled then it works after disabling it again.

image

C:\WINDOWS\system32>ping cdn.winget.microsoft.com

Pinging sni1gl.wpc.nucdn.net [2606:2800:233:1cb7:261b:1f9c:2074:3c] with 32 bytes of data:
Reply from 2606:2800:233:1cb7:261b:1f9c:2074:3c: time=11ms
Reply from 2606:2800:233:1cb7:261b:1f9c:2074:3c: time=11ms
Reply from 2606:2800:233:1cb7:261b:1f9c:2074:3c: time=11ms
Reply from 2606:2800:233:1cb7:261b:1f9c:2074:3c: time=11ms

Ping statistics for 2606:2800:233:1cb7:261b:1f9c:2074:3c:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 11ms, Maximum = 11ms, Average = 11ms

R-Adrian avatar Jan 26 '23 21:01 R-Adrian

Yes, same error...

image

BuzzwordChief avatar Jan 27 '23 08:01 BuzzwordChief