Figure out a plan to stabilize isolated extension usages

[Wyverald]

Isolated extension usages (use_extension(..., isolate=True)) are currently guarded behind the flag --experimental_isolated_extension_usages.

This feature is useful in particular as it allows an extension to easily generate repos scoped to a single user module. However, it is a rather big departure in the mental model of module extensions, as it essentially allows an extension to be evaluated "multiple times" (small print -- big simplification, etc etc); certain extensions that may rely on being "used" by its hosting module might break under this model; and the ID of an extension is not just ".bzl file label + extension name" anymore.

To stabilize this feature, we should evaluate other solutions to the "user-module-scoped repos" problem, and think about ways to mitigate the downsides of isolated extension usages (documentation? intentional limitations?).

[Wyverald]

cc @fmeum @aherrmann

[fmeum]

Just noting some insights I had that are relevant for the evaluation:

1. Isolated extension usages receive independent lockfile entries, which limits the impact of tag changes for non-deterministic extensions.
2. Without isolated extension usages, extensions may need to recommend per-module repo prefixes that include the module version if they need to distinguish module versions in the case of a multiple_version_override (e.g. because the extension relies on its own lockfile, which may conflict if multiple module versions contribute one). Nobody will realistically do this as it causes huge churn on version bumps and is only relevant in this edge case, but it may make multiple_version_override hard to use with rulesets in practice.
3. Isolated extension instances can still share a static list of deps by having the module use_repo them from the main instance.

[aherrmann]

Apologies for the late reply. It didn't get to this earlier.

I'll share how isolated extensions are currently used in rules_nixpkgs and what problem they solve.

• Only isolated extensions, or the root or core module, are allowed to directly define the name of a repo.
• The reason is that non-isolated extensions operate on a global name-space, and if any module could introduce an arbitrary name, then we could encounter name clashes. (Nix packages have too many parameters to come up with a meaningful, unique canonical name.) So, isolated extensions solve the name clash problem by introducing separate namespaces.
• Before isolated extensions were introduced, the extensions were implemented using a hub-repository approach with a custom namespace implementation. Isolated extensions simplified the implementation.
• We do have multiple extensions that need to share data and, indeed, the approach taken is to pass corresponding labels to an attribute, and have a default label that refers to a use_repo'd default in the core module.

The key issue that isolated extensions solve in this case is that of repo namespacing. Apart from that rules_nixpkgs does not further require the separation of extension instances. So, an alternative approach could be a common mechanism to introduce per client module namespaces.

Indeed, the possible presence of multiple extension instances has to be taken into account by each module extension author. For instance, take an extension providing toolchains that expects the generated toolchains repo to only be included by the core module and generates its extension metadata accordingly. If this extension is used in isolated mode, then it should expose the toolchains repo to the invoking module instead.

[alexeagle]

@aherrmann @fmeum now that we have https://github.com/bazel-contrib/bazelrc-preset.bzl as a place to recommend flag defaults which aren't yet flipped in Bazel, WDYT about setting this flag there?

[fmeum]

@alexeagle We should be careful with experimental flags. Their entire point is to have users opt into the risk of being broken if the behavior of the flag changes :-) Most of the experimental flags currently in the preset are secretly (or not so secretly, after a rename) incompatible flags, so the same concern doesn't apply to them.

[aherrmann]

cc @malt3

[malt3]

I keep thinking about this issue. It would be a useful addition in many Bazel rules. The last instance I found is rules_img, which currently doesn't offer a module extension due to the limitations @aherrmann noted above, including the problem of a global name-space in non-isolated extensions that can lead to clashes. There is no good way to form a canonical repo name that describes the identity of a container image.

I wrote down more thoughts here: https://github.com/bazel-contrib/rules_img/issues/64

The current solution I found is to exclusively depend on use_repo_rule, which is isolated but cannot share any results between modules.

[fmeum]

Would isolated extension usage help? I guess not since they can only really share a statically enumerated list of repos.

What I could see work is a version of "path mapping" for repo rules: If we somehow ensure that a repo rule doesn't actually use the rctx.attr.name attribute containing the canonical repo name (or any other way to obtain it), it could share a repo cache entry with all other repos that specify the same extension local name. The advantage is that this optimization would apply to all repos without any extra work, the downside is that only fully identical instantiations would be deduplicated - there is no way for the extension to normalize away domain-specific irrelevant differences.

[malt3]

Would isolated extension usage help?

In my specific case it would solve the name clash issue and enable the use of facts. I agree that isolated extensions would not really allow me to share repositories across modules (although an isolated extension can share data between the repositories it creates within the same module).

[fmeum]

In https://bazelbuild.slack.com/archives/C09E58X3AQ7/p1764934858912079, @malt3 and I discussed what evolved into a plan to make isolated extension usage obsolete by providing extensions with a way to implement them themselves:

1. Extend bazel mod tidy to allow extensions to suggest keyword arguments to use_repo (think my_repo = "my_module_1.2.3_my_repo"): https://github.com/bazelbuild/bazel/pull/27891
2. Extend use_repo to accept placeholders, turning the above into the more ergonomic `my_repo = "{name}_{version}_my_repo"): https://github.com/bazelbuild/bazel/pull/27890
3. Deprecate (in Bazel 9) and remove (in Bazel 10) isolated extension usages.

cc @meteorcloudy @SalmaSamy

[dzbarsky]

@fmeum Would Gazelle (for Go) also have a way to hook into the remapped names? We currently have a repo-wide go.mod that is used for most code, but there are a few projects that need a separate dependency closure. They are currently a child Bazel module with their own go.mod/go.sum and an isolated extension, so Gazelle "just works".

How could we make this kind of setup work with the proposal? I guess these specific projects could remain a child module and they would expose the same names as today, but they'd be remapped from the version-specific repos which would be shared with the parent module?

[fmeum]

@dzbarsky My plan is to add an isolated_namespace attribute to all relevant go_deps tag classes. If set to foo, use_repo arguments as generated by bazel mod tidy would take the form foo.com_github_example = "{name}_{version}_foo.com_github_example". This is what your separate module would use. We could even add a way to configure Gazelle so that it generates build files referencing these repos when run on certain subdirectories. Do you think that this would be able to replace your use of isolated extension usages?

[aherrmann]

This looks good! IIUC this effectively implements something like the hub repository pattern, but, without the need for the actual hub-repo, by having bazel mod tidy implement the mapping in-place at each use-site instead.

One difference that does seem to be worth pointing out is that while in the hub repository approach the aliases that are passed to the hub-repo function form an interface that causes breakage if changed, here (IIUC) the actual repo names, i.e. the "{name}_{version}_foo.com_github_example, form an interface that causes breakage if changed. I don't think that's objectionable, just something for module extension authors to keep in mind.