Integrating attestation verification into `brew install` (and `brew verify`)

[woodruffw]

Verification

• [X] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

I'm opening this to begin the discussion/design review process for the final piece of the build provenance work that we (@josephsweeney and I) have been doing.

For context:

1. homebrew-core has been generating attestations for a few weeks now: https://github.com/Homebrew/homebrew-core/attestations, covering an increasingly large number of "reachable" bottles
2. For pre-existing bottles that were created before the provenance rollout, we performed a one-time "backfill" attestation here: https://github.com/trailofbits/homebrew-brew-verify/attestations

Combined, this means that every currently reachable bottle in homebrew-core either has a direct attestation or a fallback "backfill" attestation.

The next step from here is to actually verify these attestations, which is what is proposed below 🙂

Proposal

TL;DR variant:

• [x] Add a BREW_VERIFY_ATTESTATIONS environment config, which will default to false during the beta period.
• [x] When BREW_VERIFY_ATTESTATIONS=true, brew install will additionally perform verification of each bottle's attestation as it gets downloaded (or reused from cache).
• [x] Regardless of BREW_VERIFY_ATTESTATIONS, the separate brew verify [formulae ...] command will allow users to explicitly verify bottle(s) by their formulae names.

The above technical elements will be implemented by myself and @josephsweeney, primarily by integrating our MVP brew verify implementation here: https://github.com/trailofbits/homebrew-brew-verify. We will make sure this implementation is suitable for both API and CLI use (i.e., so that it can be used as part of brew install's steps as well as with a stand-alone brew verify invocation).

Outstanding considerations

• How long should the beta last?
• Attestation verification currently requires the gh CLI, meaning that BREW_VERIFY_ATTESTATIONS=true will require a bootstrapping phase with brew install gh or similar to start. Should this be hash-pinned or similar?

What is the motivation for the feature?

This feature was previously reviewed and approved by the Homebrew TSC and leadership at the 2023 and 2024 AGMs.

How will the feature be relevant to at least 90% of Homebrew users?

During the beta period (i.e. BREW_VERIFY_ATTESTATIONS=false), this will have no effect on Homebrew users besides those who choose to explicitly enable it.

Once enabled by default, this will enable build provenance (modulo the backfill) verification for every bottle in homebrew-core, changing Homebrew's bottle integrity guarantee from "it matches a known digest" to "it matches a known digest and it was produced on Homebrew's own CI/CD."

What alternatives to the feature have been considered?

None.

[woodruffw]

See also #16543 for the prior phase of this 🙂

[carlocab]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

[MikeMcQuaid]

Thanks @woodruffw!

[ ] Add a BREW_VERIFY_ATTESTATIONS environment config, which will default to false during the beta period.

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

• When BREW_VERIFY_ATTESTATIONS=true, brew install will additionally perform verification of each bottle's attestation as it gets downloaded (or reused from cache).

Should probably be =1 or really set to any, non-blank value.

• Regardless of BREW_VERIFY_ATTESTATIONS, the separate brew verify [formulae ...] command will allow users to explicitly verify bottle(s) by their formulae names.

👍🏻

• How long should the beta last?

I think it's less about the time and more about just gradual ramp-up based on bugs/issues/performance regressions seen.

Generally these things go through:

• require ENV variable
• enabled by default for HOMEBREW_DEVELOPERs
• enabled by default in CI
• optional: enabled by default for HOMEBREW_DEVCMD_RUN
• enabled on master, not yet in a stable tag
• in a stable tag

• Attestation verification currently requires the gh CLI, meaning that BREW_VERIFY_ATTESTATIONS=true will require a bootstrapping phase with brew install gh or similar to start. Should this be hash-pinned or similar?

I think this is fine while the ENV variable is opt-in or for the brew verify command. I agree that making this a hard requirement for all of brew install invocations is something we probably need to resolve before this can go out to everyone as default behaviour (and probably even go to homebrew developers/CI).

Once enabled by default, this will enable build provenance (modulo the backfill) verification for every bottle in homebrew-core, changing Homebrew's bottle integrity guarantee from "it matches a known digest" to "it matches a known digest and it was produced on Homebrew's own CI/CD."

Sounds good, assuming the performance overhead here is minimal.

[Bo98]

I'm OK relying on gh as an opt-in env for brew install (and keeping brew verify in a separate tap) initially.

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

But a lot of this can be "questions for later". I was going to point out gh-attestation being closed source but I think that got fixed in yesterday's gh release? So seems like the picture is already changing. How it actually works is now clearer and I'll take a look over the code to give myself a more informed idea of what's actually involved here rather than guessing what the black box does.

[woodruffw]

How difficult would it be to support verifying attestations natively in Ruby?

I'd prefer to avoid needed gh to verify attestations, since that presents challenges to verifying the gh attestation.

I strongly agree, although unfortunately the answer to "how difficult" is currently "very difficult" -- verifying an attestation here requires a full Sigstore verifier implementation, and there currently isn't a suitable native Ruby one that's ready for use here.

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

---

Should probably default to an empty/unset variable being false rather than handling false, just FYI.

Should probably be =1 or really set to any, non-blank value.

Makes sense! I was using shorthand here; I figure this would go in Homebrew::EnvConfig, which would abstract all the 1, bool, etc. variants 🙂

---

I'm less OK with doing so unconditionally as we've generally been moving away from arbitrary external tools (beyond guaranteed system-provided tools on macOS). See past efforts of: moving from patchelf to patchelf.rb, readelf to rbelftools, the current PR for ldd and the existence of Portable Ruby. The chicken egg problem of verifying gh itself makes this more awkward and having to issue a new brew release every time gh is updated sounds annoying. A mini bundled gh (like Portable Ruby) used exclusively for this may however be possible if necessary.

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

If so, that makes sense to me (in the sense that it does an end-run around the bootstrapping problem) and IMO is an acceptable solution until we can vendor-in a pure Ruby attestation verifier.

My proposal with that:

• BREW_VERIFY_ATTESTATIONS=0 brew install: current, default behavior
• BREW_VERIFY_ATTESTATIONS=1 brew install with no gh installed: failure
• BREW_VERIFY_ATTESTATIONS=1 brew install with gh installed: performs attestation verification

[woodruffw]

Also: re: brew verify: keeping it in an external tap is OK by me if that's the consensus, but I'd prefer to upstream it (even if only as a dev-cmd) if possible: my reasoning is that brew verify will share almost exactly the same underlying implementation as the feature-flagged verification under brew install, so IMO it makes sense to de-dupe the code 🙂

[Bo98]

The closest implementation is probably the one that the RubyGems folks have been working on (they have similar native-Ruby-only requirements), but IIUC they're still in active development as well and probably aren't ready for downstream dependencies yet.

This might be an interesting watch. We don't necessarily have to move quickly here. If this route looks dead later in the year we can re-evaluate options but it seems worthwhile watching that space as a potential way forward removing the env-gate.

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

[MikeMcQuaid]

Makes sense. To align this and @carlocab's comment: is the thought that users would have to explicitly brew install gh first, only after which BREW_VERIFY_ATTESTATIONS=1 brew install ... would be effective?

I think it'd be OK for it to install gh itself (we have a one-liner for such things), it's more just that it can't really be default until it's entirely in Ruby.

[Bo98]

even if only as a dev-cmd

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

[woodruffw]

If RubyGems needs/likes any input/contributions from me then I'm happy to do that too.

Paging @segiddins 🙂

---

I'd be ok as a dev-cmd if we make it clear in the command description too that it's under development. We can promote it to cmd when we're ready to remove the env-gate.

Sounds good! We'll make that explicit.

[segiddins]

https://github.com/segiddins/sigstore-cosign-verify should almost be usable!

[woodruffw]

I've opened #17049 with the first bits of this: that PR adds HOMEBREW_VERIFY_ATTESTATIONS and directs brew update to bootstrap gh if HOMEBREW_VERIFY_ATTESTATIONS is set. Once set, brew install verifies the attestations for any bottles being fetched from homebrew-core.

[carlocab]

#17049 seems to work well, but verifying attestations seems a bit slow:

❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &> /dev/null; )  0.85s user 0.27s system 37% cpu 2.982 total
❯ brew uninstall hello
Uninstalling /opt/homebrew/Cellar/hello/2.12.1... (11 files, 177KB)
❯ time (HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &>/dev/null)
( HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &> /dev/null; )  0.62s user 0.19s system 79% cpu 1.018 total

Note: bottle pre-fetched so download time is not included

[woodruffw]

seems to work well, but verifying attestations seems a bit slow:

Yeah, I think there's some low hanging fruit that I can optimize here: gh attestation verify is probably pulling more things down the pipe than it strictly needs to, and there may be optimizations that it can further do internally. I'll do some profiling.

[carlocab]

FYI, trying to run brew upgrade when gh is outdated and HOMEBREW_VERIFY_ATTESTATIONS is set results in:

==> Upgrading gh
  2.47.0 -> 2.48.0
==> Verifying attestation for gh
Error: The bottle for gh has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/opt/homebrew/bin/gh attestation verify /Users/carlocab/Library/Caches/Homebrew/downloads/51240e810aca942d2fdc644c6f72d02b0c6cf00ea4e2653417b3ca959c427762--gh--2.48.0.arm64_sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json --cert-identity https://github.com/trailofbits/homebrew-brew-verify/.github/workflows/backfill_signatures.yml@refs/heads/main` exited with 127. Here's the output:

This is likely because /opt/homebrew/bin/gh either doesn't exist or is a broken symlink at the point we tried to execute it.

[woodruffw]

Thanks for catching that! Looking at it now.

Edit: I was able to reproduce this:

==> Upgrading gh
  2.47.0 -> 2.48.0 
==> Verifying attestation for gh
Error: The bottle for gh has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/opt/homebrew/bin/gh attestation verify /Users/william/Library/Caches/Homebrew/downloads/51240e810aca942d2fdc644c6f72d02b0c6cf00ea4e2653417b3ca959c427762--gh--2.48.0.arm64_sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json --cert-identity https://github.com/trailofbits/homebrew-brew-verify/.github/workflows/backfill_signatures.yml@refs/heads/main` exited with 127. Here's the output:

[woodruffw]

@carlocab I think your theory is right: upgrade unlinks each previous keg right before installation, meaning that the symlink is probably broken or nonexistent. If I remember my formula building blocks correctly, I'm pretty sure this means that we need to reference the gh executable via its cellar path rather than its default prefix linkage. Trying that now.

Edit: Opened #17106 with a prospective fix.

[woodruffw]

Updates:

• https://github.com/Homebrew/brew/pull/17220 increases the reliability of our gh invocation by pulling credentials from a few other well-known sources. There are additional tricks we can adopt here that I'll follow up on.
• gh attestation verify itself should be >33% faster after the latest gh release, thanks to some optimizations done by the GH package security team. In practice that means ~500ms savings for brew install:

arbet:~ william$ time (HOMEBREW_VERIFY_ATTESTATIONS=1 brew install hello &>/dev/null)

real	0m2.530s
user	0m1.160s
sys	0m0.393s

arbet:~ william$ time (HOMEBREW_VERIFY_ATTESTATIONS= brew install hello &>/dev/null)

real	0m1.956s
user	0m0.978s
sys	0m0.326s

(This puts us within ~600ms of non-verifying brew install, and I think we should target ~300ms for GA.)

[woodruffw]

A variant of the bootstrapping bug is when the user does HOMEBREW_VERIFY_ATTESTATIONS=1 brew install gh without gh already being installed:

$ HOMEBREW_VERIFY_ATTESTATIONS=1 brew install gh
==> Downloading https://ghcr.io/v2/homebrew/core/gh/manifests/2.49.2
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/e21907ac557448cf904309f43a204664563ea1eeec33647d923ea3b308dc6902--gh-2.49.2.bottle_manifest.json
==> Fetching gh
==> Downloading https://ghcr.io/v2/homebrew/core/gh/blobs/sha256:a9b943de9c581a59
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/1d47281ddb66d5bfb053e1f74cff84446c6626f87d7536b93b5c7dbc5555f073--gh--2.49.2.arm64_sonoma.bottle.tar.gz
==> Verifying attestation for gh
==> Installing `gh`...
==> Downloading https://ghcr.io/v2/homebrew/core/gh/manifests/2.49.2
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/e21907ac557448cf904309f43a204664563ea1eeec33647d923ea3b308dc6902--gh-2.49.2.bottle_manifest.json
==> Fetching gh
==> Downloading https://ghcr.io/v2/homebrew/core/gh/blobs/sha256:a9b943de9c581a59
Already downloaded: /Users/william/Library/Caches/Homebrew/downloads/1d47281ddb66d5bfb053e1f74cff84446c6626f87d7536b93b5c7dbc5555f073--gh--2.49.2.arm64_sonoma.bottle.tar.gz
Error: Operation already in progress for gh.formula
Another active Homebrew process is already using gh.formula.
Please wait for it to finish or terminate it to continue.
Error: Failure while executing; `/opt/homebrew/bin/brew install --formula gh` exited with 1.

The fix should be straightforward here: we shouldn't bootstrap gh with ensure_formula! when the user is explicitly installing gh for the first time.

Edit: For recordkeeping purposes: I'll fix this today.

[nikaro]

Is it supposed to work with custom Tap?

• I edited my publish workflow to generate build attestations: https://github.com/nikaro/homebrew-tap/commit/3062e1d724df1212ae30e5058a71c3b91771841f
• It seems to work: https://github.com/nikaro/homebrew-tap/actions/runs/9127733861
• I can see attestations here: https://github.com/nikaro/homebrew-tap/attestations

But when i install my Formula it doesn't do the verification:

➤ env | grep HOMEBREW
HOMEBREW_VERIFY_ATTESTATIONS=1
➤ brew install vim-language-server
==> Auto-updating Homebrew...
Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with
HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).
==> Fetching nikaro/tap/vim-language-server
==> Downloading https://github.com/nikaro/homebrew-tap/releases/download/vim-language-server-2.3.1_2/vim-language-server-2.3.1_2.a
Already downloaded: /Users/nicolas/Library/Caches/Homebrew/downloads/61a8dbd873b61a440c78a1dd6ec5ad114fb3c6cc73f6dead262d4dff24c984db--vim-language-server-2.3.1_2.arm64_sonoma.bottle.tar.gz
==> Installing vim-language-server from nikaro/tap
==> Pouring vim-language-server-2.3.1_2.arm64_sonoma.bottle.tar.gz
🍺  /opt/homebrew/Cellar/vim-language-server/2.3.1_2: 18 files, 4.6MB
==> Running `brew cleanup vim-language-server`...
Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.
Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

I am missing something or doing something wrong?

[woodruffw]

Is it supposed to work with custom Tap?

• I edited my publish workflow to generate build attestations: https://github.com/nikaro/homebrew-tap/commit/3062e1d724df1212ae30e5058a71c3b91771841f

• It seems to work: https://github.com/nikaro/homebrew-tap/actions/runs/9127733861

• I can see attestations here: https://github.com/nikaro/homebrew-tap/attestations

But when i install my Formula it doesn't do the verification:

➤ env | grep HOMEBREW

HOMEBREW_VERIFY_ATTESTATIONS=1

➤ brew install vim-language-server

==> Auto-updating Homebrew...

Adjust how often this is run with HOMEBREW_AUTO_UPDATE_SECS or disable with

HOMEBREW_NO_AUTO_UPDATE. Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

==> Fetching nikaro/tap/vim-language-server

==> Downloading https://github.com/nikaro/homebrew-tap/releases/download/vim-language-server-2.3.1_2/vim-language-server-2.3.1_2.a

Already downloaded: /Users/nicolas/Library/Caches/Homebrew/downloads/61a8dbd873b61a440c78a1dd6ec5ad114fb3c6cc73f6dead262d4dff24c984db--vim-language-server-2.3.1_2.arm64_sonoma.bottle.tar.gz

==> Installing vim-language-server from nikaro/tap

==> Pouring vim-language-server-2.3.1_2.arm64_sonoma.bottle.tar.gz

🍺  /opt/homebrew/Cellar/vim-language-server/2.3.1_2: 18 files, 4.6MB

==> Running `brew cleanup vim-language-server`...

Disable this behaviour by setting HOMEBREW_NO_INSTALL_CLEANUP.

Hide these hints with HOMEBREW_NO_ENV_HINTS (see `man brew`).

I am missing something or doing something wrong?

You're not missing anything, it just isn't supported yet 🙂. We need some way for 3p taps to communicate that they're fully attested, which in effect will probably require a new piece of tap metadata.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[woodruffw]

Not stale, still tracking remaining items.

[woodruffw]

#17692 expands the current verification beta to include people with developer flags set.

[louis993546]

Hi, just FYI if user have signed in to GitHub Enterprise on gh, attestation will fail.

==> Upgrading xz
  5.4.6 -> 5.6.2 
==> Verifying attestation for xz
Error: The bottle for xz has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/usr/bin/env GH_TOKEN=****** /opt/homebrew/opt/gh/bin/gh attestation verify $(my homebrew cache location)/downloads/$(some kind of hash)--xz--5.6.2.arm64_sonoma.bottle.tar.gz --repo trailofbits/homebrew-brew-verify --format json` exited with 1. Here's the output:
An unsupported host was detected. Note that gh attestation does not currently support GHES

[Bo98]

That particular issue should have already been fixed. Let me know which version you're running (brew --version).

[louis993546]

@Bo98 unfortunately the log was from last week and I have since logged out and homebrew could have been updated since then. I have tried to reinstall xz and everything seems to be working fine now. Thanks!

[DomT4]

@woodruffw Just as an FYI more than anything, it turns out if you have an older gh in your $PATH it'll use that one and may not support the feature, causing the process to fall apart. Unsure if worth version checking or enforcing the Homebrew version?

==> Verifying attestation for ca-certificates
ln -s ../Cellar/ca-certificates/2024-03-11/share/ca-certificates ca-certificates
Error: The bottle for ca-certificates has an invalid build provenance attestation.

This may indicate that the bottle was not produced by the expected
tap, or was maliciously inserted into the expected tap's bottle
storage.

Additional context:

attestation verification failed: Failure while executing; `/usr/bin/env GH_TOKEN=****** GH_HOST=github.com /Users/Dominyk/go/bin/gh attestation verify /opt/homebrew/var/homebrewcache/downloads/9af01538d558e40dd9cf236c9e9d04f265bd12b0e10d9ee3881ca667e043acd9--ca-certificates--2024-07-02.all.bottle.tar.gz --repo Homebrew/homebrew-core --format json` exited with 1. Here's the output:
unknown command "attestation" for "gh"

Usage:  gh <command> <subcommand> [flags]

Available commands:
  alias
  api
  auth
  browse
  cache
  co
  codespace
  completion
  config
  extension
  gist
  gpg-key
  issue
  label
  org
  pr
  project
  release
  repo
  ruleset
  run
  search
  secret
  ssh-key
  status
  variable
  workflow

[woodruffw]

Just as an FYI more than anything, it turns out if you have an older gh in your $PATH it'll use that one and may not support the feature, causing the process to fall apart. Unsure if worth version checking or enforcing the Homebrew version?

Thanks -- I have https://github.com/Homebrew/brew/pull/17926 open to specialize the error in that case, but it popped off my stack 😅 -- I'll set aside some time tomorrow to address the comments there.

[DomT4]

@woodruffw Thanks! Bo's comment seems sound, I don't know if there's many of us kicking around with a custom gh. Couldn't even tell you why I was using a non-brew one at this point in all honesty!