Create module extensions

[aherrmann]

Related to #181

Is your feature request related to a problem? Please describe.

The new bzlmod dependency management system introduces a new concept called module extensions. rules_nixpkgs currently only exposes repository rules. To fully "modularize" rules_nixpkgs we need to expose the same functionality that is currently exposed as repository rules in the form of module extensions.

Describe the solution you'd like

Bazel's repository rules are evaluated individually. I.e. each nixpkgs_package is evaluated separately and creates its own external repository. Module extensions on the other hand are evaluated collectively. Each invocation across all MODULE.bazel files in a given project and its transitive dependencies generates a tag with some metadata. In the end the corresponding module extension is evaluated with all these tags as inputs and can then use this information and invoke external tools to generate repository rule invocations to import all required external dependencies.

A direct translation of the current repository rules into a module extension could look something like this on the use-site:

bazel_dep(name = "rules_nixpkgs_core", version = "1.0")
nixpkgs = use_extension("@rules_nixpkgs_core//:extensions.bzl", "nixpkgs")

nixpkgs.git_repository(name = "nixpkgs", revision = "12.11", sha256 = ...)
nixpkgs.package(name = "boost", attribute_path = "boost175", repository = "nixpkgs")

bazel_dep(name = "rules_nixpkgs_cc", version = "1.0")
nixpkgs_cc = use_extension("@rules_nixpkgs_cc//:extensions.bzl", "nixpkgs_cc")
nixpkgs_cc.cc_configure(name = "nixpkgs_config_cc", repository = "nixpkgs")
use_repo(nixpkgs_cc, "nixpkgs_config_cc")

module(
    name = "user_project",
    ...
    toolchains_to_register = ["@nixpkgs_config_cc_toolchains//:toolchain"],
)

Note, the above is only a sketch and may not be possible in exactly that form.

Describe alternatives you've considered

The sketch above is a direct translation of the features provided by rules_nipxkgs right now. However, due to the collective evaluation of module extensions with bzlmod, we could go further and also import transitive Nix dependencies into Bazel in a way that handles overlapping dependency graphs correctly. E.g. if a project imports two C libraries libA and libB from nixpkgs and both of these libraries depend on libC, then the module extension could generate imports of all three libA, libB, and libC and expose the dependency graph to Bazel. E.g.

# MODULE.bazel
bazel_dep(name = "rules_nixpkgs_cc", version = "1.0")
nixpkgs_cc = use_extension("@rules_nixpkgs_cc//:extensions.bzl", "nixpkgs_cc")
nixpkgs_cc.cc_library(name = "libA", attribute_path = "libA", repository = "nixpkgs")
nixpkgs_cc.cc_library(name = "libB", attribute_path = "libB", repository = "nixpkgs")
# @libA//:BUILD.bazel
cc_library(name = "libA", srcs = ["lib/libA.so"], deps = ["@libC//:libC"])
# @libB//:BUILD.bazel
cc_library(name = "libB", srcs = ["lib/libB.so"], deps = ["@libC//:libC"])
# @libC//:BUILD.bazel
cc_library(name = "libC", srcs = ["lib/libC.so"])

Again, the above is only a sketch and may not be possible in exactly that form.

However, I would propose to start with a direct translation and consider something more elaborate like the above as a later extension.

Additional context

The design document in the section "Design: Module rules and non-Bazel registries" explains module extensions in more detail.

Tasks

• [ ] Design module extensions
  • [x] Nix repository imports
  • [x] Nix package imports
  • [ ] toolchain imports
• [ ] Implement module extensions
  • [ ] nix_repo to import Nix repositories
    • [x] default to use a global default repo
    • [x] github to import a Github repository
    • [x] http to download a repository
    • [x] file to import from a local file
    • [x] expr to import from a Nix expression
    • [x] override to override a global default repo
    • [x] Handle nixopts
    • [ ] Support location expansion
  • [ ] nix_pkg to import Nix packages
    • [x] attr to use a globally unified Nix attribute path
    • [x] local_attr to import a Nix package by attribute path locally
    • [x] local_file to import from a Nix file locally
    • [x] local_expr to import from a Nix expression locally
  • [ ] toolchains
    • [x] #480
    • [ ] ...
• [ ] Documentation
  • [ ] Clarify distinction between module extension nix_pkg and hub repo accessor nix_pkg (same for nix_repo).

### Tasks
    - [x] #480
- [ ] https://github.com/tweag/rules_nixpkgs/issues/506

[aherrmann]

However, due to the collective evaluation of module extensions with bzlmod, we could go further and also import transitive Nix dependencies into Bazel in a way that handles overlapping dependency graphs correctly.

This is somewhat related to https://github.com/tweag/rules_nixpkgs/issues/49 which proposes a way to generated separate nixpkgs_package imports for a set of Nix packages.

[tshaynik]

I'm interested to start working on this.

[aherrmann]

I'm interested to start working on this.

@tshaynik Thank you, that would be a valuable contribution. The upcoming Bazel 6 release is targeted to use bzlmod by default, so it would be good to have bzlmod support for rules_nixpkgs soon.

The wider context in https://github.com/tweag/rules_nixpkgs/issues/181 will probably also be useful for this task.

[ar3s3ru]

Hi folks, sorry to chime in, just a question regarding this, as I'd love to start using rules_nixpkgs but I'm currently on Bzlmod.

I currently don't see a way to import a package from a local flake.nix(https://github.com/tweag/rules_nixpkgs/blob/9b13ed0469116a37658040a7b62d70aea90ad0bc/core/nixpkgs.bzl#L813), not in the current extensions nor in the checklist above. Is that unplanned functionality at the moment, or is there a different way to use that?

Alternatively, is it possible to mix both MODULE.bazel (mainly to fetch @rules_nixpkgs_core) and nixpkgs_flake_pacakge/toolchains in WORKSPACE.bzlmod?

[aherrmann]

Thanks for reaching out @ar3s3ru! Flake support is also planned but tracked in https://github.com/tweag/rules_nixpkgs/issues/348.

Yes, it's possible to mix MODULE.bazel and the old repo rules. There are several ways to do this. WORKSPACE.bzlmod is one, though one has to be a bit careful about which repos are visible in which context. Other options are Bazel's new use_repo_rule feature, which lets you call repo rules directly in MODULE.bazel, or an ad-hoc module extension as done here for example.