Split bottling and testing a formula into separate steps

[carlocab]

Currently, we build bottles and test these bottles all in one test-bot call. Most of the code for this call lives in lib/tests/formulae.rb.

I suggest we split this up so that this can be done in separate steps in a GitHub workflow:

1. Build bottles, then upload them.
2. Download the built bottles, and then test them.

I see at least three benefits from doing this:

1. We can build :all bottles on the Linux runner and then test them on our macOS runners.
2. We can use a different runner/container when building bottles and testing the bottles and their dependents. This is useful for formulae built in the Debian Wheezy container: https://github.com/Homebrew/homebrew-test-bot/blob/6a82b403363b623ec67b97b001d3bedf188fd684/lib/tests/formulae_dependents.rb#L21-L24
3. We'd be in a much better place for parallelising dependent testing, at least on Linux. We could spin up several runners that would download the built bottles and test disjoint subsets of the dependent formulae. (This probably requires splitting up the dependency calculations from formulae_dependents.rb. This could potentially be done in the bottle building step.)

The last benefit is pretty important: I think we're going to start regularly hitting the three-day workflow time limit on Linux fairly soon given that dependency trees tend to much larger on Linux.

[carlocab]

CC @Homebrew/maintainers for thoughts, and in case someone else wants to pick this up. (Please feel free.)

[MikeMcQuaid]

I suggest we split this up so that this can be done in separate steps in a GitHub workflow:

1. Build bottles, then upload them.
2. Download the built bottles, and then test them.

I strongly agree. Some previous splits: https://github.com/Homebrew/homebrew-test-bot/pull/582 https://github.com/Homebrew/homebrew-test-bot/pull/635

Some other useful splits:

• run brew audit --online and brew fetch only on Linux (or at least: on Linux first)

• We can build :all bottles on the Linux runner and then test them on our macOS runners.

I presume you mean "only on the Linux runner" here. I'm 👍🏻 either way but that would allow us to save some macOS build time.

3. We'd be in a much better place for parallelising dependent testing, at least on Linux. We could spin up several runners that would download the built bottles and test disjoint subsets of the dependent formulae. (This probably requires splitting up the dependency calculations from formulae_dependents.rb. This could potentially be done in the bottle building step.)

Even on macOS I think this would make things better as we could split dependent testing into many jobs rather than having them all happen in a single, long-running one.

We could also potentially do some smart caching where we cache the uploaded bottles based on their respective contents (rather than Git commit) SHAs to avoid repeatedly rebuilding the same bottle in the case that e.g. we need to revision bump some dependents.

[danielnachun]

I am wondering if, as a starting point, we should make a test-dependent workflow which is given the names of the dependent and its dependency, the platform to test, and a pull request number from which to download the bottle for the dependency. test-bot would then dispatch a job for each dependent on each platform and those jobs would just be added to the queue.

Given that we don't normally build dependents from source, on Linux the default ubuntu-latest runner should always be sufficient to pour the bottle and test it. In fact we could probably reduce the usage of the self-hosted runner substantially because only a few formulae on Linux need more than 6 hours just to build the bottle (there are also a few that need extra disk space). As for macOS we'd probably also benefit from not tying up a runner for 3 days for a single job as well, and if macOS runners ever became ephemeral, we'd get all the same benefits as on Linux.

[MikeMcQuaid]

I am wondering if, as a starting point, we should make a test-dependent workflow which is given the names of the dependent and its dependency, the platform to test, and a pull request number from which to download the bottle for the dependency. test-bot would then dispatch a job for each dependent on each platform and those jobs would just be added to the queue.

I like the idea of doing this 👍🏻

As for macOS we'd probably also benefit from not tying up a runner for 3 days for a single job as well, and if macOS runners ever became ephemeral, we'd get all the same benefits as on Linux.

I think this is probably worth holding off until we have some ephemeral macOS runners.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[MikeMcQuaid]

We could also potentially do some smart caching where we cache the uploaded bottles based on their respective contents (rather than Git commit) SHAs to avoid repeatedly rebuilding the same bottle in the case that e.g. we need to revision bump some dependents.

I think this would make a huge difference, particularly in cases like boost.

For example, we would use git log -1 --pretty=format:%H Formula/boost.rb as the cache key here (assuming boost changed in the PR) and if we've got a bottle that was already built on that commit hash for the boost formula: we install it from the bottle instead of building it from source again.

If we wanted the cache key to be resistant to merges/rebases: we could instead use git hash-object Formula/boost.rb.

[carlocab]

I think one piece of low-hanging fruit to get this going would be to teach test-bot to check the working directory for a bottle for one of the @testing_formulae, and install that bottle whenever required.

This may require keeping track of the dependency tree of each formula in @testing_formulae since it's relatively common for these formulae to depend on each other in a given test-bot run.

If this step makes sense, I could get a PR for this up soon.

One difficulty I see with caching across different runs: where do we cache the bottles? My current understanding of the way GitHub Actions works is that re-running a workflow discards artefacts built from a previous run, so we may need some sort of external cache to store these things in. This introduces the problem of managing access to this external cache.

[iMichka]

We can use our google cloud account. Credentials need to be tweaked but are already in place. We could use a storage there.

[MikeMcQuaid]

I think one piece of low-hanging fruit to get this going would be to teach test-bot to check the working directory for a bottle for one of the @testing_formulae, and install that bottle whenever required.

This may require keeping track of the dependency tree of each formula in @testing_formulae since it's relatively common for these formulae to depend on each other in a given test-bot run.

If this step makes sense, I could get a PR for this up soon.

Good idea, I like it.

One difficulty I see with caching across different runs: where do we cache the bottles? My current understanding of the way GitHub Actions works is that re-running a workflow discards artefacts built from a previous run, so we may need some sort of external cache to store these things in. This introduces the problem of managing access to this external cache.

My suggestion/thought was that we use https://github.com/actions/cache for this.

The tricky bit will be deciding how/what to cache. I'd suggest we cache each bottle based on the SHA-1 of the formula file.

[MikeMcQuaid]

My suggestion/thought was that we use https://github.com/actions/cache for this.

The tricky bit will be deciding how/what to cache. I'd suggest we cache each bottle based on the SHA-1 of the formula file.

Elaborating on this: I think we will probably want to use the PR number + bottle_tag for the actions cache key so we can share these as appropriate. Inside test-bot itself, I think we'll need to handle the "was this bottle made based on the correct version of the formula or not" caching; I don't think we can really do this easily inside the actions logic itself.

[carlocab]

Ah, I had forgotten about actions/cache. That does sound look a good use for it.

I'll start work on implementing bottle installs in test-bot. Note: I don't intend (yet) to implement validation of the bottle to be installed. I think this will depend on the caching scheme we eventually settle on.

[MikeMcQuaid]

Sounds good @carlocab 👍🏻

[carlocab]

Thanks for this @MikeMcQuaid!