Extremely slow download of large files from GitHub Releases in winget (compared to downloading same file in browser)

[Jaifroid]

Brief description of your issue

I find the download speed of a large 1.29GB package in winget is orders of magnitude slower than downloading the same package in the browser.

Steps to reproduce

Run winget install kiwix.wikimed.electron in Terminal. Download for me is around 500Kb per second. It would take around 45 minutes to completely download the file, if not longer. In any case I didn't wait around, it was too painful.

When I click on the download link provided in winget, and open it in my browser, on the same machine I get a download speed of 6.5Mb per second.

I have experienced this on multiple occasions on two different machines running on the same network. Winget is unusable for downloading this >1GB package.

Expected behavior

I expect similar download speeds to browser.

Actual behavior

See above -- painfully slow.

Environment

Windows Package Manager (Preview) v1.3.692-preview
Copyright (c) Microsoft Corporation. All rights reserved.

Windows: Windows.Desktop v10.0.22000.613
Package: Microsoft.DesktopAppInstaller v1.18.692.0

[Trenly]

Have you tried changing which downloader is used in the settings?

https://github.com/microsoft/winget-cli/blob/2937bdfcb3681c45a53eabc6cc45fafc327fa956/schemas/JSON/settings/settings.schema.0.2.json#L139-L152

[Jaifroid]

Thank you for the suggestion. I was not aware of this setting. I'll try it.

[jedieaston]

I've been seeing this in the last few days too, although I haven't troubleshooted (figured my connection had a blip). Is there something going on with Delivery Optimization and GitHub right now?

[denelon]

I'm not aware of any issues. Is it possible Windows Updates or something else is running in the background affecting the throughput for Delivery Optimization?

[Jaifroid]

Well I've just done another test with the same file, and through winget the file downloaded 32MB (of a 1.29GB file) in around 40 seconds. So, without stopping winget, I ctrl-clicked the link that winget provides to the original paciage, I accepted the download in my browser, and the same file downloads over 200MB in around 6 seconds, while winget is still going at a snail's pace. My PC is fully up to date, no updates being downloaded.

Is it the PowerShell download process? Is it single-threaded? There's something very odd going on. It's clearly not my local computer or connection. It used to be much faster than this in winget.

[JohnMcPMS]

I've just hit the same thing, and the wininet downloader seems as fast as ever, while DO (aka default) is 🐌 (and not Turbo)

Going to follow up with the DO team to see if they can help interpret the logs that I've collected.

[JohnMcPMS]

Apparently the issue is DO getting timeouts from github's CDNs, resulting in DO thinking the maximum throughput is very low. They are considering how future code can better handle this, but there isn't a lot we can do to fix this in winget right now.

Hopefully the service issues will resolve and things will go back to normal. If not, we can consider putting in something like "if github, don't use DO". Or maybe "if DO takes over X seconds to download Y MB, give up and fall back to wininet".

The immediate options are:

1. Use wininet as the downloader; it is slower than DO because it is a very basic implementation, but since it isn't trying to be smart it isn't as affected by externalities.
2. Just try again; CTRL+C, Up, Enter. If you get lucky, the CDN doesn't hiccup.
3. According to the DO team, you can set Absolute bandwidth limits to prevent the throughput detection issue. a. Open Settings (🪟 + i ) b. Find Delivery Optimization advanced options c. Select Absolute bandwidth d. Check Limit how much bandwidth is used for downloading updates in the foreground e. Set to whatever limit you want (100 Mbps?)

[Jaifroid]

That's very helpful. Thanks for investigating and following up. I'll try one of these solutions tomorrow (it's late in my time zone), and will report back.

[Jaifroid]

I confirm that changing the network setting to "wininet" produces a much more acceptable download speed. Perhaps this should be an automatic fallback (or one of the other solutions if easier to implement as a fallback). I'm not sure how one would detect that DO is being artificially rate-limited, however.

[hueldoeu]

what means DO? i try to winget upgrade libreoffice and it's superslow

[Masamune3210]

Delivery Optimization, it's the system that powers Windows Update and some other things

[hueldoeu]

Delivery Optimization, it's the system that powers Windows Update and some other things

what can i do to increase cmd => winget upgrade libreoffice ? someone told something about wininet. i know that winget bases on the idea of apt-get and in apt-get i could easily choose the mirror on aptitude. every download was super fast

[Masamune3210]

Honestly, at least specifically with libreoffice, this might just be a issue of slow servers and no mirror. Downloading it specifically on my system is slow with either backend. We can't do much about servers being slow

[hueldoeu]

Honestly, at least specifically with libreoffice, this might just be a issue of slow servers and no mirror. Downloading it specifically on my system is slow with either backend. We can't do much about servers being slow

you are right. i copied the link displayed in cmd into ms edge and downloaded it from there. speed 4-9 KB/s. ehem. that's ... slow ... the link was https://downloadarchive.documentfoundation.org/libreoffice/old/7.3.4.2/win/x86_64/LibreOffice_7.3.4.2_Win_x64.msi

[Jaifroid]

This issue persists, implying it's basic to the DO and GitHub combination, and really needs a fallback put in place on the winget end. I'm only posting this now because I got a new PC, and was wondering why winget download performance had become so terrible on this spanking new laptop. Then I remembered this thread. On my old PC I had set winget to use wininet and had never looked back. To notice the (huge) difference, you need to attempt to install a large (>200MB) installer, like Kiwix.WikiMed.Electron.

[hueldoeu]

i think it work it works this way: when a new version of libreoffice comes out, many upgrade automatically because they wrote a.script or soemthing lile that. i know 2 tools (gui) where you can force winget upgrade automatically every startup. because of that (must be lots of windows 10/11 gadgets) download gets very slow. then, when most of these soulless bots have libreoffice installed, its time for the sane, civilized people to upgrade their winget list. now the download speed of libreodfoce in its newest version (7.3.4 which exists since 9th of june 2022) is higher and acceptable. so statisticslly after a new version, you have to wait 1 month, which is ok for libreoffice because of all the program winget upgrades, it only happens with libreoffice that the download speed is low and updates for libreoffice arent that important to install immediately.

This issue persists, implying it's basic to the DO and GitHub combination, and really needs a fallback put in place on the winget end. I'm only posting this now because I got a new PC, and was wondering why winget download performance had become so terrible on this spanking new laptop. Then I remembered this thread. On my old PC I had set winget to use wininet and had never looked back. To notice the (huge) difference, you need to attempt to install a large (>200MB) installer, like Kiwix.WikiMed.Electron.

[Jaifroid]

@hueldoeu Thanks for that explanation, but in my case I am installing-testing my app Kiwix.WikiMed.Electron, not libreoffice, and my app has a minute user base in comparison. There is an ongoing issue regarding the interaction between winget's use of Delivery Optimization and GitHub, at least for large installers (it's not likely to be very noticeable for installers < 70MB).

[JohnMcPMS]

This issue persists, implying it's basic to the DO and GitHub combination, and really needs a fallback put in place on the winget end.

My previous comment contains the mitigations that one can use, and the only suggestion that I can think of that we could do. But even that (which is effectively, "is DO achieving some arbitrary minimum bandwidth?") isn't foolproof, if you are truly bandwidth limited then we would be moving to the potentially slower option.

I'm not sure that there is really a good, automatic way for us to do a better job than a team that is dedicated to doing this work. The right solution to my mind is to drive fixes into either DO or github or both.

[Jaifroid]

1. Use wininet as the downloader; it is slower than DO because it is a very basic implementation, but since it isn't trying to be smart it isn't as affected by externalities.
2. Just try again; CTRL+C, Up, Enter. If you get lucky, the CDN doesn't hiccup.
3. According to the DO team, you can set Absolute bandwidth limits to prevent the throughput detection issue. a. Open Settings (🪟 + i ) b. Find Delivery Optimization advanced options c. Select Absolute bandwidth d. Check Limit how much bandwidth is used for downloading updates in the foreground e. Set to whatever limit you want (100 Mbps?)

@JohnMcPMS Unfortunately, mitigations 1 and 3 are no longer working. At least on Windows 11 22H2 mitigation 3 no longer seems to make any difference to the ridiculously slow download of large installers from GitHub when DO is enabled in winget. By large, I'm referring to installers > 500MB. The one that affects me most, because I have to fight for hours to get it validated, is Kiwix.WikiMed, which is 1.2GB. I can literally wait an hour while winget plods its way through the download using DO, yet if I direct-download the installer file in my browser, it takes about 3 minutes.

And with the latest winget, wininet download (mitigation 1) seems to fail at the hash checking stage for large installers too. It fails with:

An unexpected error occurred while executing the command:
0x80072efd : unknown error

Screenshot below shows tests I did with wininet enabled in the winget settings. The first shows it failing with a local manifest of the latest 1.2GB WikiMed release that I was trying to validate for release on winget. The second is a small installer which succeeds. The last is with the current WikiMed package on winget. All installers are hosted on GitHub.

I guess no-one tests winget with large downloads, and slow download speeds won't make much difference for a 50MB installer. The experience is pretty much unusable with large installers, and none of the workarounds are working any more...

[airtonix]

Apparently the issue is DO getting timeouts from github's CDNs, resulting in DO thinking the maximum throughput is very low. They are considering how future code can better handle this, but there isn't a lot we can do to fix this in winget right now.

Hopefully the service issues will resolve and things will go back to normal. If not, we can consider putting in something like "if github, don't use DO". Or maybe "if DO takes over X seconds to download Y MB, give up and fall back to wininet".

The immediate options are:

1. Use `wininet` as the downloader; it is slower than DO because it is a very basic implementation, but since it isn't trying to be smart it isn't as affected by externalities.

2. Just try again; CTRL+C, Up, Enter.  If you get lucky, the CDN doesn't hiccup.

3. According to the DO team, you can set `Absolute bandwidth` limits to prevent the throughput detection issue.
   a. Open Settings (🪟 + i )
   b. Find `Delivery Optimization advanced options`
   c. Select `Absolute bandwidth`
   d. Check `Limit how much bandwidth is used for downloading updates in the foreground`
   e. Set to whatever limit you want (100 Mbps?)

Anyone know the powershell one-liner to set this without having to use mouse or keyboard (I'm applying this via automation)

edit: some quick research:

• https://learn.microsoft.com/en-us/windows/deployment/do/waas-delivery-optimization-reference
• https://learn.microsoft.com/en-us/windows/deployment/do/waas-delivery-optimization-setup
• https://2pintsoftware.com/delivery-optimization-powershell-cmdlets/

ugh... i can't find anything that lets me modify the above settings.

I'm going to revert to just writing my own winget.json into: %LOCALAPPDATA%\Packages\Microsoft.DesktopAppInstaller_8wekyb3d8bbwe\LocalState\settings.json with something like :

{
    "installBehavior": {
        "preferences": {
            "scope": "user"
        },
        "telemetry": {
            "disable": true
        },
        "network": {
            "downloader": "wininet"
        }
    }
}

values found here: https://github.com/microsoft/winget-cli/blob/master/doc/Settings.md

[Karl-WE]

As libreoffice is often stated this is a related PR #2310

As 2310 is closed I am fine to pick up the help-wanted of this PR to get the mysterious problem solved.

[Jaifroid]

@Karl-WE As developer of large apps (1.29GB), this issue is still frustrating me. I get very very slow download with Kiwix.WikiMed for example (hosted on GitHub). Here is a log of my installation of this app just now. It took 37 minutes to download 1.29GB, a download that normally takes no more than 5 or 6 minutes over my broadband.

2023-01-07 11:13:01.285 [CLI ] Failed to get msix signature hash, fall back to direct download.
2023-01-07 11:13:01.301 [CLI ] Generated temp download path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] Downloading to path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] DeliveryOptimization downloading from url: https://github.com/kiwix/kiwix-js-windows/releases/download/v2.2.62-WikiMed/KiwixWebAppWikiMed_2.2.62.0_AnyCPU.appxbundle
2023-01-07 11:48:27.275 [CORE] DeliveryOptimization error: 0x80d02002, extended error: 0x80190191
2023-01-07 11:48:27.276 [FAIL] D:\a\_work\1\s\external\pkg\src\AppInstallerCommonCore\DODownloader.cpp(289)\WindowsPackageManager.dll!00007FFB0EEB28F9: (caller: 00007FFB0EEB31C0) Exception(2) tid(29e4) 80D02002 
2023-01-07 11:48:27.410 [CORE] Started applying motw to C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a with zone: 3
2023-01-07 11:48:27.419 [CORE] Finished applying motw
2023-01-07 11:48:27.420 [CORE] WinINet downloading from url: https://github.com/kiwix/kiwix-js-windows/releases/download/v2.2.62-WikiMed/KiwixWebAppWikiMed_2.2.62.0_AnyCPU.appxbundle
2023-01-07 11:50:44.857 [CORE] Download hash: 035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:50:44.857 [CORE] Download completed.
2023-01-07 11:50:44.858 [CLI ] Installer hash verified
2023-01-07 11:50:44.859 [CORE] Started applying motw to C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a with zone: 2
2023-01-07 11:50:44.872 [CORE] Finished applying motw
2023-01-07 11:50:44.874 [CLI ] Successfully renamed downloaded installer. Path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\KiwixWebAppWikiMed_2.2.62.0_AnyCPU.msix
2023-01-07 11:50:44.985 [CORE] Starting StagePackageAsync operation #0: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\KiwixWebAppWikiMed_2.2.62.0_AnyCPU.msix
2023-01-07 11:50:45.044 [CORE] Begin waiting for operation #0
2023-01-07 11:50:45.044 [CORE] Begin blocking for operation #0
2023-01-07 11:50:55.808 [CORE] Successfully completed #0
2023-01-07 11:50:55.808 [CORE] Starting RegisterPackageByFullNameAsync operation #1: Kiwix.WikiMed_2.2.62.0_neutral_~_qwjmsdmcnwx12
2023-01-07 11:50:55.823 [CORE] Begin waiting for operation #1
2023-01-07 11:50:55.823 [CORE] Begin blocking for operation #1
2023-01-07 11:50:56.290 [CORE] Successfully completed #1

[hueldoeu]

@Karl-WE As developer of large apps (1.29GB), this issue is still frustrating me. I get very very slow download with Kiwix.WikiMed for example (hosted on GitHub). Here is a log of my installation of this app just now. It took 37 minutes to download 1.29GB, a download that normally takes no more than 5 or 6 minutes over my broadband.

2023-01-07 11:13:01.285 [CLI ] Failed to get msix signature hash, fall back to direct download.
2023-01-07 11:13:01.301 [CLI ] Generated temp download path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] Downloading to path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] DeliveryOptimization downloading from url: https://github.com/kiwix/kiwix-js-windows/releases/download/v2.2.62-WikiMed/KiwixWebAppWikiMed_2.2.62.0_AnyCPU.appxbundle
2023-01-07 11:48:27.275 [CORE] DeliveryOptimization error: 0x80d02002, extended error: 0x80190191
2023-01-07 11:48:27.276 [FAIL] D:\a\_work\1\s\external\pkg\src\AppInstallerCommonCore\DODownloader.cpp(289)\WindowsPackageManager.dll!00007FFB0EEB28F9: (caller: 00007FFB0EEB31C0) Exception(2) tid(29e4) 80D02002 
2023-01-07 11:48:27.410 [CORE] Started applying motw to C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a with zone: 3
2023-01-07 11:48:27.419 [CORE] Finished applying motw
2023-01-07 11:48:27.420 [CORE] WinINet downloading from url: https://github.com/kiwix/kiwix-js-windows/releases/download/v2.2.62-WikiMed/KiwixWebAppWikiMed_2.2.62.0_AnyCPU.appxbundle
2023-01-07 11:50:44.857 [CORE] Download hash: 035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:50:44.857 [CORE] Download completed.
2023-01-07 11:50:44.858 [CLI ] Installer hash verified
2023-01-07 11:50:44.859 [CORE] Started applying motw to C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a with zone: 2
2023-01-07 11:50:44.872 [CORE] Finished applying motw
2023-01-07 11:50:44.874 [CLI ] Successfully renamed downloaded installer. Path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\KiwixWebAppWikiMed_2.2.62.0_AnyCPU.msix
2023-01-07 11:50:44.985 [CORE] Starting StagePackageAsync operation #0: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\KiwixWebAppWikiMed_2.2.62.0_AnyCPU.msix
2023-01-07 11:50:45.044 [CORE] Begin waiting for operation #0
2023-01-07 11:50:45.044 [CORE] Begin blocking for operation #0
2023-01-07 11:50:55.808 [CORE] Successfully completed #0
2023-01-07 11:50:55.808 [CORE] Starting RegisterPackageByFullNameAsync operation #1: Kiwix.WikiMed_2.2.62.0_neutral_~_qwjmsdmcnwx12
2023-01-07 11:50:55.823 [CORE] Begin waiting for operation #1
2023-01-07 11:50:55.823 [CORE] Begin blocking for operation #1
2023-01-07 11:50:56.290 [CORE] Successfully completed #1

i'm ok with the speed, could be faster, but i don't mind. at lest winget works so good that it shows me WHICH of my apps need an update. sometimes teamviewer doesn't work because it's updating in the background and winget doesn't tell me there is an update. i don't have to update immediately but it's good to have an overview if there is an update for edge, telegram (store version requires microsoft account), libreoffice, vlc, anydesk, ....

[Karl-WE]

I am not declining your point it still exists and is only partly reproducible. Are you from EU? @ItzLevvie did a very in depht troubleshooting waiting for her instructions what could be helpful as a repro and what logs are needed from wireshark or fiddler.

[hueldoeu]

yes germany (EU). winget is like apt-get or aptitude and i really need this feature. it's very helpful.

[ItzLevvie]

Try and see if https://github.com/microsoft/winget-pkgs/pull/93256 & https://github.com/microsoft/winget-pkgs/pull/93255 makes a difference. This removes the streaming install from the latest version of both packages.

If it does work then that's great but just an FYI that winget-create and YamlCreate.ps1 will automatically add this back.

~~Unfortunately it's hard for some people to reproduce due to different configurations & locations. I'm in EU West - London (UK) and I was able to download Kiwix.WikiMed successfully in less than 5 minutes with 130 Mbps regardless if the manifest has SignatureSha256 or not.~~

^ EDIT: By reading previous comments — it fails on Delivery Optimization which I haven't tried but does work on fine on WinINet.

Download with SignatureSha256 (WinINet) = less than 5 minutes Download without SignatureSha256 (WinINet) = less than 5 minutes

There were a few times where WinGet did do a fallback to direct download because it failed to use SignatureSha256. I checked logs each time I downloaded WikiMed and it still managed to download & install in less than 5 minutes.

I took a look at your logs and

2023-01-07 11:13:01.285 [CLI ] Failed to get msix signature hash, fall back to direct download.
2023-01-07 11:13:01.301 [CLI ] Generated temp download path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] Downloading to path: C:\Users\gkant\AppData\Local\Temp\WinGet\Kiwix.WikiMed.2.2.62.0\035080d826d851505358ea653aa0caafe4ab655f8d8ce9aaebe0eaec2319932a
2023-01-07 11:13:01.302 [CORE] DeliveryOptimization downloading from url: https://github.com/kiwix/kiwix-js-windows/releases/download/v2.2.62-WikiMed/KiwixWebAppWikiMed_2.2.62.0_AnyCPU.appxbundle
2023-01-07 11:48:27.275 [CORE] DeliveryOptimization error: 0x80d02002, extended error: 0x80190191
2023-01-07 11:48:27.276 [FAIL] D:\a\_work\1\s\external\pkg\src\AppInstallerCommonCore\DODownloader.cpp(289)\WindowsPackageManager.dll!00007FFB0EEB28F9: (caller: 00007FFB0EEB31C0) Exception(2) tid(29e4) 80D02002 

WinGet did a streaming install and failed; then spent 35 minutes doing a download with Delivery Optimization and failed [1]; so WinGet ended up doing a download with WinINet and managed to install WikiMed.

[1] If your WinGet downloader is set to default (which auto chooses between Delivery Optimization / WinINet?) then change it to be wininet so that it doesn't try to use Delivery Optimization which is where most of the time is spent.

EDIT 2: Fixed a few issues. Turns out 35 minutes is spent on Delivery Optimization instead of streaming install so the SignatureSha256 probably doesn't make much difference here. Setting the downloader to always use WinINet should cut down that 35 minutes.

[Jaifroid]

I'm from the UK, and I mostly get these very slow speeds. I'm the developer of the WikiMed and Wikivoyage apps, which are very large because of the compressed copies of Wikipedia data contained in them, so I test installation of them a lot. Until very recently installation was completely broken with winget for large appx UWP apps due to a signature verification problem (signature was valid, but winget couldn't get if from GitHub and errored out). Now, you can see from the logs that the signature verification still can't be got, but instead winget defaults to downloading the app directly and verifying the signature after download, which is much better behaviour.

However, the speed is the issue under discussion here. Comparing winget install and Microsoft Store install of the same app (but different packages, because the winget one is on GitHub and the MS Store one is on the MS Store), the MS Store is way way faster, 5 or 6 mins versus 37 mins with the test logged above.

This isn't a criticism of winget. The problem is in the interaction between winget and GitHub servers. However, if I direct-download the exact same package from GitHub in my browser, the download is normal speed (5 or 6 mins, rarely exceeding 10 mins for 1.29GB). To me, this implies that winget could speed things up considerably by defaulting to a simple PowerShell Invoke-WebRequest command! The problem seems to come from the attempt to use Delivery Optimization. I think DO is highly acceptable for @hueldoeu's use-case, when wanting to do background updates of a number of apps, but if you want to download an app and use it straight away, it's frustratingly slow and is going to put people off from using winget (which I think is a great service, by the way -- I just want to improve the UX).

[Jaifroid]

@ItzLevvie thanks for that analysis! I'll take a look at those PRs.

[hueldoeu]

@Jaifroid correct

[Karl-WE]

Just got noticed about a new release of Winget @Jaifroid does it still repro with libreoffice or your packages?