Improve JSON API for Install Performance Improvements

[Rylan12]

Verification

• [x] This issue's title and/or description do not reference a single formula e.g. brew install wget. If they do, open an issue at https://github.com/Homebrew/homebrew-core/issues/new/choose instead.

Provide a detailed description of the proposed feature

The formula.json and cask.json files are downloaded almost every time an install or upgrade happens, and these files are large.

• [x] Shard the JSON files
  • Have a different file for each supported OS, so e.g. formula.arm64_sequoia.json and cask.arm64_sequoia.json
  • https://github.com/Homebrew/brew/pull/20038
  • https://github.com/Homebrew/brew/pull/20045
  • https://github.com/Homebrew/formulae.brew.sh/pull/1905
• [x] Create a new JSON API that contains the minimum information required to download a bottle's manifest file (name, version, hash)
  • https://github.com/Homebrew/brew/pull/20051
• [ ] Store formula API information in the bottle manifest file
• [x] When installing a formula, use the new lightweight API and retrieve all missing information needed from the manifest
• [ ] When installing a cask, use the new lightweight API to check whether a new version is needed, and then retrieve the remaining info from the cask's individual JSON file and/or the Ruby file
  • Include a marker in the JSON for whether the ruby file is needed or not
• [x] Move tap migrations and other similar config files to live here in a single JSON file
• [ ] Create a new JSON API file to store formula/cask descriptions that’s downloaded on-demand for brew info or brew search --desc —-eval-all

What is the motivation for the feature?

To make installing formulae and casks via the API faster

How will the feature be relevant to at least 90% of Homebrew users?

Increased performance for all API installs

What alternatives to the feature have been considered?

• Use a git repo with just the JSON files for each comparison (and since there are only 2 files it shouldn't have the issues we had with homebrew/core)
• Using BSON and Webpack but they are (including when gzip compressed) larger than gzip compressed JSON

[apainintheneck]

I like this approach and I'm excited to see what this will look like. This seems like an alternative approach to https://github.com/Homebrew/brew/issues/16410 which maybe should get closed in favor of this one. I did some work on that at the time that got abandoned so feel free to change, use or remove any of the existing logic.

[MikeMcQuaid]

This seems like an alternative approach to #16410 which maybe should get closed in favor of this one. I did some work on that at the time that got abandoned so feel free to change, use or remove any of the existing logic.

Agreed, thanks. Yeh, we should remove the JSON v3 logic at some point before closing this issue.

[apainintheneck]

Okay, I'll get on removing the JSON v3 logic and I'll tag @Rylan12 to make sure I don't remove anything you plan on using.

[EricFromCanada]

This could also be useful for improving the formulae.brew.sh load time by, say, fetching JSON of just names + versions + descriptions and rendering it client-side with pagination.

[SukkaW]

Don't know if this is the right place to put the feedback, but I have enabled $HOMEBREW_USE_INTERNAL_API, and on the first run (I was trying to do brew upgrade, and the only outdated thing is fastfetch), it appears that Homebrew tries to bootstrap for using the new internal API and downloaded shit loads of small JSON.

In the end, the Homebrew took 55 seconds to finish this first brew upgrade and upgraded one bottle fastfetch.

Don't know if this only happens on the first time (as a bootstrap) or will always happen on every brew upgrade, but IMHO the new shard JSON could use some parallel downloading as well.

[itfaqtradersn-lab]

G