FR: Share extracted external libraries across workspaces

[dan-cohn]

ATTENTION! Please read and follow:

• if this is a question about how to build / test / query / deploy using Bazel, or a discussion starter, send it to [email protected]
• if this is a bug or feature request, fill the form below as best as you can.

Description of the problem / feature request:

I would like Bazel to be able to share extracted repos across workspaces. In other words, the "repository cache" should go deeper than simply making downloaded archives available in a centralized location. It would help save time and space if the unpacked contents were also shared.

Feature requests: what underlying problem are you trying to solve with this feature?

Bazel's repository cache is shared across workspaces, which avoids downloading the same external libraries multiple times. This saves a lot of time and network utilization, especially in CI environments where builds occur across many different repos and versions. The problem is that only the original downloaded artifacts are shared and not the extracted files. This means that Bazel has to repeatedly unzip/unpack archives. This takes a lot of time, consumes CPU, and fills the filesystem with multiple copies of the exact same extracted libraries.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

N/A

What operating system are you running Bazel on?

RHEL 7/8 and CentOS 7/8

What's the output of bazel info release?

release 3.4.1

If bazel info release returns "development version" or "(@non-git)", tell us how you built Bazel.

N/A

What's the output of git remote get-url origin ; git rev-parse master ; git rev-parse HEAD ?

N/A

Have you found anything relevant by searching the web?

No.

Any other information, logs, or outputs that you want to share?

N/A

[Wyverald]

Hi dan-cohn, we're working on a new design for Bazel's external dependencies, and this idea is being incorporated into the new design. Please consider joining the [email protected] mailing list or following along on the #external-deps channel on Slack.

[tim-chaplin-dd]

Hi dan-cohn, we're working on a new design for Bazel's external dependencies, and this idea is being incorporated into the new design. Please consider joining the [email protected] mailing list or following along on the #external-deps channel on Slack.

Hello from the future. It sounds like this new design ended up being Bzlmod, but from what I can tell it doesn't address the problem described here since the unpacked repos still live in the external subdir alongside the ones declared in WORKSPACE. Please let me know if I'm wrong about that.

[Wyverald]

I'm currently working on a design for this. It's trickier than expected so it's taking a bit more time than I'd like, but hopefully I'll be able to share it this week.

[Wyverald]

Update: unfortunately the doc is still not quite ready, and I'll be on leave next week. So this will need to wait until September.

[Wyverald]

The design is ready for review here: https://docs.google.com/document/d/1ZScqiIQi9l7_8eikbsGI-rjupdbCI7wgm1RYD76FJfM/edit

It's, uh, a bit longer than I expected.

[LittleCuteBug]

Hi, is there any update on this feature?

[iancha1992]

@bazel-io fork 7.1.0

[Wyverald]

Too late for 7.2.0; punting to later.