extract: Leave a hint in place of the removed bottles block

[Kentzo]

• [x] Have you followed the guidelines in our Contributing document?
• [x] Have you checked to ensure there aren't other open Pull Requests for the same change?
• [x] Have you added an explanation of what your changes do and why you'd like us to include them?
• [x] Have you written new tests for your changes? Here's an example.
• [ ] Have you successfully run brew style with your changes locally?
• [ ] Have you successfully run brew typecheck with your changes locally?
• [ ] Have you successfully run brew tests with your changes locally?

---

[Kentzo]

@MikeMcQuaid Would you be interested in the PR adding the --extract-bottles-dir= to support extracting bottles of the homebrew/core formulae via skopeo such that they can be easily re-published later into GitHub Packages of another repo?

[MikeMcQuaid]

@Kentzo can you explain the workflow of how that would work? Bottles are built for a specific formula name/prefix/keg and cannot be used with another name/prefix/keg.

[Kentzo]

I'm writing a script that extracts a formula of the given version alongside its dependencies (recursively, at versions that correspond to the git revision of the formula) to a custom tap. And I want to archive the corresponding bottles as well.

Homebrew already knows how to download bottles regardless of root_url (or rather custom storages are already compatible with Homebrew). Homebrew also knows how to upload bottles to Github Packages. Thus it should be feasible to to provide an API to extract bottles to either a local directory or a Github Packages repo.

With that feature implemented my script could look like:

$archived = []
$formulae = [{'name': ..., 'version': ..., 'revision': ...}]
for $name, $version, $revision in $formulae:
    if $name in $archived:
        continue

    if $version:
        brew extract $name my/repo --version=$version --extract-bottles-dir=<gh-packages-or-local-dir>
        $revision = # parse revision from the stdout
    else
        brew extract $name my/repo --git-revision=$revision --extract-bottles-dir=<gh-packages-or-local-dir>
        $version = # parse version from the stdout

    $archived.append($name)

    brew deps my/repo/$name@$version
    $deps = #parse deps from stdout
    formulae.extend([{'name': $d['name'], 'revision': $revision} for $d in $deps])

[MikeMcQuaid]

@Kentzo The problem is: brew extract and extracted bottles use different names so the two outputs are not compatible.

[Kentzo]

Not sure I'm following. I'm not offering to extract them separately, i.e. extract formula first then extract bottles next. But rather both being part of the single invocation.

[MikeMcQuaid]

@Kentzo That's my point exactly: they cannot be part of the same invocation. This is why we don't extract bottles.

Extracting a formula extracts to [email protected] which installs to Cellar/formula@version/full_version.

The bottle, however, is from formula.rb which installs to Cellar/formula/full_version.

You can't use a bottle from the latter for the prior. It installs to the wrong location.

[Kentzo]

I’m not sure where user’s Cellar comes into play here. I suggest the bottles to be taken from the homebrew/core repo on Github Packages.

[Kentzo]

Here is a particular example:

brew extract sqlite my/repo --version=3.44.2 creates a new formula at /opt/homebrew/Library/Taps/my/homebrew-repo/[email protected] with contents identical to https://github.com/Homebrew/homebrew-core/blob/1c9fed5481c9a62c5314b887b41e31f14edd8e53/Formula/s/sqlite.rb except for the bottles do end being stripped.

Bottles for that revision are still available at https://github.com/homebrew/homebrew-core/pkgs/container/core%2Fsqlite/151970181?tag=3.44.2 which is also available via REST API at docker://ghcr.io/homebrew/core/sqlite:3.44.2 (e.g. using skopeo).

[gromgit]

brew extract sqlite my/repo --version=3.44.2 creates a new formula at /opt/homebrew/Library/Taps/my/homebrew-repo/[email protected] with contents identical to https://github.com/Homebrew/homebrew-core/blob/1c9fed5481c9a62c5314b887b41e31f14edd8e53/Formula/s/sqlite.rb except for the bottles do end being stripped.

Are you then renaming the formula back to sqlite.rb? (You'd also have to change the class name inside that file, of course.) If you don't, you'll likely find that manually adding the original bottle block back into your new formula won't work quite the way you expect, because:

1. the bottle unpacks to .../Cellar/sqlite/..., while the versioned formula expects .../Cellar/[email protected]/...
2. the bottled files may have hardcoded references to .../Cellar/sqlite/..., which would have to be fixed up before things can work

After you rename the formula, though, Homebrew now sees two formulae called sqlite, which may lead to other problems (still processing my first coffee of the day, so I don't have further details on this).

[Kentzo]

@gromgit Thank you for the clarification. In my mind I was keeping the formula name intact but adjusting bottle's path on Github Packages instead. However, I did not verify that behavior yet. Let me try something material and I will get back with the results.

[MikeMcQuaid]

In my mind I was keeping the formula name intact

If you do this: this version cannot be installed at the same time as the same version from other taps (e.g. core) and will be brew upgraded and brew cleanupd away.

If this isn't a problem for you: that's fine, good luck; but it's something we'd rather not support in Homebrew/brew because of these sharp edges.

[Kentzo]

@MikeMcQuaid

Just to clarify, by “intact” I meant [email protected], i.e. as created by brew extract.

Does the above still apply?

[MikeMcQuaid]

@Kentzo Yes. Unless you're extracting as a sqlite formula: you cannot use the bottles from an older sqlite version.

[Kentzo]

In other words homebrew/core/sqlite and my/repo/sqlite can co-exist (installed at the same time), but homebrew/core/sqlite and my/repo/[email protected] cannot. And the reason for this is that compatibility: old version of the same formula needs to exist at .../Cellar/sqlite/3.44.2, as if it were installed some time ago when 3.44.2 was the current version of sqlite.

[MikeMcQuaid]

In other words homebrew/core/sqlite and my/repo/sqlite can co-exist (installed at the same time)

Not quite. They cannot both be installed at the same time but can both share bottles.

but homebrew/core/sqlite and my/repo/[email protected] cannot

These can be installed at the same time but cannot share bottles.

And the reason for this is

The layout of the bottles. Extract one and you'll see.

[Kentzo]

The layout of the bottles. Extract one and you'll see.

I see now.

Is there a way to keep versioned name for the formula (so I can have both [email protected] and [email protected] at the same time in the tap) but otherwise make brew have as if it was just sqlite.rb?

[MikeMcQuaid]

@Kentzo Nope.

[Kentzo]

@MikeMcQuaid Just to confirm, there is currently no interest in having the --archive option either to extract the unmodified formula, exactly as found (i.e. without renaming and stripping bottles)?

[MikeMcQuaid]

@Kentzo I guess it just doesn't seem very useful to me to have an option that will be broken by default/design?