Remove `homebrew-cask`'s dependency on `raw.githubusercontent.com`

[stevapple]

Verification

• [X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

Move homebrew-cask away from directly depending on raw.githubusercontent.com. If possible, move to a mechanism that's possible for rsync for mirroring.

What is the motivation for the feature?

Since homebrew/cask tap was fully deprecated, it becomes hard for people to use brew cask when raw.githubusercontent.com is unavailable or blocked by ISP. More specifically, we always see errors like this:

Error: docker: Failed to download resource "docker.rb"
Failure while executing; `/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.1.3\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.5\)\ curl/8.1.2 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://raw.githubusercontent.com/Homebrew/homebrew-cask/b237285b6df4e1ce951ab0665bebc134e64a7fa8/Casks/docker.rb` exited with 7. Here's the output:
curl: (7) Failed to connect to raw.githubusercontent.com port 443 after 3180 ms: Couldn't connect to server

For formulae.brew.sh we can use a mirror, but there's no way to mirror raw.githubusercontent.com as a local repository.

How will the feature be relevant to at least 90% of Homebrew users?

It's at least relevant to 90% of Homebrew users in China😔

What alternatives to the feature have been considered?

Just don't hard-code such mechanism. If possible, provide a way that's possible for rsync.

Another solution to not move away from GitHub is to use GitHub REST API instead.

[osalbahr]

Since homebrew/cask tap was fully deprecated

I’m not sure that’s the case. As far as I know it’s just not the default anymore. Does setting HOMEBREW_NO_INSTALL_FROM_API help?

Example:

HOMEBREW_NO_INSTALL_FROM_API=1 brew install --cask <cask_name>

[MikeMcQuaid]

Support already exists for mirroring here: https://github.com/Homebrew/brew/blob/41ca77c6ac571d58eda505d38cce3685d2b3d818/Library/Homebrew/api/cask.rb#L32-L36

The command to generate it is brew generate-cask-api: https://github.com/Homebrew/brew/blob/41ca77c6ac571d58eda505d38cce3685d2b3d818/Library/Homebrew/dev-cmd/generate-cask-api.rb#L71

And you can see the GitHub Actions pipeline for formulae.brew.sh which hosts it here: https://github.com/Homebrew/formulae.brew.sh/blob/ab277a45c581ea6b09f0f0807cf673a846d8f192/.github/workflows/scheduled.yml#L38-L50

I'd recommend setting up your own mirror for all of formulae.brew.sh and using that.

[stevapple]

@MikeMcQuaid Thank for your information. However, setting up the mirror for cask-source still doesn’t work because HomeBrew will always send a HEAD request to raw.githubusercontent.com even the mirror has the resource. We need to get rid of such unnecessary cause of failure.

[MikeMcQuaid]

@stevapple It will try the authoritative source first because it's more secure to do so and fall back to the mirror if it fails. I believe this is consistent with how we handle similar mirrors in China?

[stevapple]

The problem is, if the HEAD (not the one to GET manifest content) request fails (time out / connection down) HomeBrew will crash immediately. It should fall back to the mirror silently, or, given sometimes it’s a long time to wait, use API at the first place.

[MikeMcQuaid]

It should fall back to the mirror silently

Agreed.

@stevapple can you provide the full output of a relevant brew command failing with --debug and --verbose?

[stevapple]

$ brew upgrade zoom --cask --verbose --debug
==> Downloading https://mirrors.ustc.edu.cn/homebrew-bottles/api/cask.jws.json
/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --user-agent Homebrew/4.1.4\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.5\)\ curl/8.1.2 --header Accept-Language:\ en --fail --remote-time --output /Users/yr.chen/Library/Caches/Homebrew/api/cask.jws.json --location --time-cond /Users/yr.chen/Library/Caches/Homebrew/api/cask.jws.json --compressed --speed-limit 100 --speed-time 5 https://mirrors.ustc.edu.cn/homebrew-bottles/api/cask.jws.json
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  987k  100  987k    0     0  1731k      0 --:--:-- --:--:-- --:--:-- 1765k
/usr/local/Homebrew/Library/Homebrew/brew.rb (Cask::CaskLoader::FromAPILoader): loading zoom
==> Upgrading 1 outdated package:
/usr/local/Homebrew/Library/Homebrew/brew.rb (Cask::CaskLoader::FromPathLoader): loading /usr/local/Caskroom/zoom/.metadata/5.15.5.20753/20230726134711.484/Casks/zoom.rb
zoom 5.15.5.20753 -> 5.15.6.21146
==> Started upgrade process for Cask zoom
==> Upgrading zoom
/usr/local/Homebrew/Library/Homebrew/brew.rb (Cask::CaskLoader::FromAPILoader): loading zoom-for-it-admins
==> Printing caveats
==> Cask::Installer#fetch
/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.1.4\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.5\)\ curl/8.1.2 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://raw.githubusercontent.com/Homebrew/homebrew-cask/0ce6a97a5ccd8d5095c411711856b5b9a073ac1e/Casks/zoom.rb
==> Purging files for version 5.15.6.21146 of Cask zoom
Error: zoom: Failed to download resource "zoom.rb"
Failure while executing; `/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.1.4\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.5\)\ curl/8.1.2 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://raw.githubusercontent.com/Homebrew/homebrew-cask/0ce6a97a5ccd8d5095c411711856b5b9a073ac1e/Casks/zoom.rb` exited with 7. Here's the output:
curl: (7) Failed to connect to raw.githubusercontent.com port 443 after 17 ms: Couldn't connect to server


/usr/local/Homebrew/Library/Homebrew/system_command.rb:313:in `assert_success!'
/usr/local/Homebrew/Library/Homebrew/utils/curl.rb:232:in `block in curl_headers'
/usr/local/Homebrew/Library/Homebrew/utils/curl.rb:210:in `each'
/usr/local/Homebrew/Library/Homebrew/utils/curl.rb:210:in `curl_headers'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:483:in `resolve_url_basename_time_file_size'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:475:in `resolved_url_and_basename'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:323:in `resolved_basename'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:307:in `cached_location'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:280:in `temporary_path'
/usr/local/Homebrew/Library/Homebrew/download_strategy.rb:404:in `fetch'
/usr/local/Homebrew/Library/Homebrew/downloadable.rb:87:in `fetch'
/usr/local/Homebrew/Library/Homebrew/api/cask.rb:39:in `source_download'
/usr/local/Homebrew/Library/Homebrew/cask/installer.rb:581:in `load_cask_from_source_api!'
/usr/local/Homebrew/Library/Homebrew/cask/installer.rb:66:in `fetch'
/usr/local/Homebrew/Library/Homebrew/cask/upgrade.rb:193:in `upgrade_cask'
/usr/local/Homebrew/Library/Homebrew/cask/upgrade.rb:114:in `block in upgrade_casks'
/usr/local/Homebrew/Library/Homebrew/cask/upgrade.rb:113:in `each'
/usr/local/Homebrew/Library/Homebrew/cask/upgrade.rb:113:in `upgrade_casks'
/usr/local/Homebrew/Library/Homebrew/cmd/upgrade.rb:253:in `upgrade_outdated_casks'
/usr/local/Homebrew/Library/Homebrew/cmd/upgrade.rb:137:in `upgrade'
/usr/local/Homebrew/Library/Homebrew/brew.rb:94:in `<main>'

[Bo98]

Yes mirrors are currently broken and have been for a while: #15169. (JWS mirroring is unaffected due to its special handling)

The curl_headers didn't used to error but it now does, and this is incorrect. It probably changed for some other callers, so the fix is probably adding a rescue ErrorDuringExecution to the download strategy.

Support already exists for mirroring here:

This doesn't support arbitrary mirroring like you suggest as it uses HOMEBREW_API_DEFAULT_DOMAIN. With that said, adding arbitrary mirrors should be safer now we have checksum validation so we probably don't need to pin to default domains.

I however do not recommend mirroring cask-source due to race conditions that will cause checksum failures. It was an emergency mechanism that has not scaled well with the security enhancements we have made since then (a trade off we accepted at the time due to it being a last resort mirror), and it seems like no one has actually needed it for quite some time anyway, as mirroring to it has been broken for 4 months. A proper mirror should mirror directly from git and produce URLs with the commit hash in them, keeping previous ones for at least an hour or so, depending on what HOMEBREW_API_AUTO_UPDATE_SECS values (+ the time it takes to run a long brew upgrade/install) you are willing to support/afford hosting. If we add configurable mirror support, we should require this URL format.

We could probably improve cask-source ourselves, though there's not been much reason to spend time on it given it's not exactly been highly used anymore.

[MikeMcQuaid]

A proper mirror should mirror directly from git and produce URLs with the commit hash in them, keeping previous ones for at least an hour or so, depending on what HOMEBREW_API_AUTO_UPDATE_SECS values (+ the time it takes to run a long brew upgrade/install) you are willing to support/afford hosting. If we add configurable mirror support, we should require this URL format.

@Bo98 This is likely too involved for any non-maintainer to reliably build.

---

I think the fix here is to fix up curl_headers accordingly and nudge people towards cask-source for now. The race conditions there are likely a better outcome than the known failures elsewhere.

[Bo98]

@Bo98 This is likely too involved for any non-maintainer to reliably build.

Depends on the setup. In a scenario of mirroring every 15 minutes, you can simplify this a lot to simply keeping the last 4 versions. You don't need to serve anything between if it's the same mirror as the JWS, and you could probably do it without git as a result if you get the commit hash from any JSON.

Though in any case, we don't even support mirrors here atm anyway so simple or not, it's not possible for any third-party mirrors to do anything currently.

If cask-source was actually used, we probably would have already done this ourselves by now.

[carlocab]

On a slightly related note, the current behaviour of always issuing --head requests (which we currently short-circuit when downloading bottles) makes some accesses to HOMEBREW_CACHE really slow, because brew still makes the --head request even when the wanted resource has already been cached. See, for, example:

==> Fetching git
==> Downloading https://mirrors.edge.kernel.org/pub/software/scm/git/git-htmldocs-2.41.0.tar.xz
/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.1.4-31-g6e1293b\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.4.1\)\ curl/7.88.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head https://mirrors.edge.kernel.org/pub/software/scm/git/git-htmldocs-2.41.0.tar.xz
/usr/bin/env /usr/local/Homebrew/Library/Homebrew/shims/shared/curl --disable --cookie /dev/null --globoff --show-error --user-agent Homebrew/4.1.4-31-g6e1293b\ \(Macintosh\;\ Intel\ Mac\ OS\ X\ 13.4.1\)\ curl/7.88.1 --header Accept-Language:\ en --retry 3 --fail --location --silent --head --request GET https://mirrors.edge.kernel.org/pub/software/scm/git/git-htmldocs-2.41.0.tar.xz
Already downloaded: /Users/carlocab/Library/Caches/Homebrew/downloads/2f91121f03a39bc2438e7e0fe9cd9458169ed4b5f9f9e89ce708a4d9c7bbb325--git-htmldocs-2.41.0.tar.xz

It also looks like two --head requests are being made? I'm not sure why we should be doing that.

[Bo98]

In certain cases some of these steps are intentional (e.g. :latest casks on bad servers).

I noticed the speed of it like you though and did shortcut the bottle case. You should see near instant processing times for those.

[MikeMcQuaid]

Depends on the setup. In a scenario of mirroring every 15 minutes, you can simplify this a lot to simply keeping the last 4 versions. You don't need to serve anything between if it's the same mirror as the JWS, and you could probably do it without git as a result if you get the commit hash from any JSON.

Sorry, I mean the actual commands and knowing what should be output when/how/where/etc.

It also looks like two --head requests are being made? I'm not sure why we should be doing that.

This seems wrong, I agree.

In certain cases some of these steps are intentional (e.g. :latest casks on bad servers).

Yeh, it seems like this behaviour should be opt-in to that case rather than default for all cases, though CC @reitermarkus.

[carlocab]

I noticed the speed of it like you though and did shortcut the bottle case. You should see near instant processing times for those.

Yep, I had noticed this quite a while ago but hadn't gotten around to looking into it.

I have a launchctl job that does something like brew outdated | xargs brew fetch to avoid waiting for things to download. However, the extra head requests still made brew upgrade really slow (sometimes it seemed like brew spent more time making head requests than it did actually pouring bottles).

Yeh, it seems like this behaviour should be opt-in to that case rather than default for all cases, though CC @reitermarkus.

Agreed.

[MikeMcQuaid]

However, the extra head requests still made brew upgrade really slow (sometimes it seemed like brew spent more time making head requests than it did actually pouring bottles).

Ooof, that's bad 😬

[Bo98]

I have a launchctl job that does something like brew outdated | xargs brew fetch to avoid waiting for things to download. However, the extra head requests still made brew upgrade really slow (sometimes it seemed like brew spent more time making head requests than it did actually pouring bottles).

There should be zero head requests with brew fetch for GitHub Packages bottles. I tested this now and that remains the case after my fix.

If you're upgrading things built from source (or non-GHP bottles): yes it's still an issue, but that should be rarer and BFS is generally slow anyway, and you also lose things like the post_install optimisation. And Casks too I guess, but that's a trickier problem to solve as we can't optimise based on how we know one server (ghcr.io) works like we do for bottles (e.g. for bottles, we don't need remote modification time as we assume blobs are immutable, and we don't need remote filename as ghcr.io doesn't offer it and we don't need it).

There's definitely room for improvement, but the Homebrew/core bottle situation now is likely unimprovable, or not in a significant way, as I've already short circuited almost all the code and has runtime in the order of milliseconds (+ your download speed if not cached).

[carlocab]

Sorry, to be clear: I was describing the behaviour of brew upgrade with cached downloads before the bottle download fix. Now the fetch steps happen instantaneously. It'd be nice to limit the head requests to cases where we know they're needed, instead of doing them for all (non-bottle) downloads.

[MikeMcQuaid]

It'd be nice to limit the head requests to cases where we know they're needed, instead of doing them for all (non-bottle) downloads.

Agreed.

[Trilok6085]

Support already exists for mirroring here:

https://github.com/Homebrew/brew/blob/41ca77c6ac571d58eda505d38cce3685d2b3d818/Library/Homebrew/api/cask.rb#L32-L36

The command to generate it is brew generate-cask-api:

https://github.com/Homebrew/brew/blob/41ca77c6ac571d58eda505d38cce3685d2b3d818/Library/Homebrew/dev-cmd/generate-cask-api.rb#L71

And you can see the GitHub Actions pipeline for formulae.brew.sh which hosts it here: https://github.com/Homebrew/formulae.brew.sh/blob/ab277a45c581ea6b09f0f0807cf673a846d8f192/.github/workflows/scheduled.yml#L38-L50

I'd recommend setting up your own mirror for all of formulae.brew.sh and using that.

Vs/true complaint accept