Rule to generate docker images with nix store paths

[jcpetruzza]

Context: I'm currently using bazel with remote execution using this branch; it's not an ideal solution, but it's a first step and there could be other people interested in trying it out.

The goal was to have a docker image that contains all the nix store paths that occur in the workspace, so that the remote executors also inherit them. For this, I'm introducing a repository rule nixpkgs_docker_image, that I'm then using in my workspace as follows:

nixpkgs_docker_image(
    name = "nix_deps_image",
    srcs = [
        "@cc_toolchain_nixpkgs_info//nixpkg",
        "@nixpkgs_python_toolchain_python3//nixpkg",
        "@nixpkgs_sh_posix_config//nixpkg",
        "@rules_haskell_ghc_nixpkgs//nixpkg",
        "@nixpkgs_valgrind//nixpkg",
    ],
    bazel = "@nixpkgs_bazel//nixpkg",
    repositories = {"nixpkgs": "@nixpkgs"},
)

In srcs, one should include every usage of the nixpkgs_package rule, either explicit (as @nixpkgs_valgrind in this example) or implicit as those coming from the toolchain rules. One then gets two targets:

• @nix_deps_image//:image: a genrule that uses dockerTools.buildLayeredImage to build a tarball
• @nix_deps_image//:stream: a sh_binary that uses dockerTools.streamLayeredImage to output the tarball on stdout.

One then needs to ensure that the docker image is available before starting a remote build, either by manually pushing the image, or running a CI step.

All this relies on the nixpkgs_package rule creating a nixpkg subpackage in the package repository, containing the exact nix-expression used by nix-build to create the package. Ideally, this would all be more transparent to the user, who would just mention the toolchain and the package in srcs and the rule would figure out the rest, but for a proof-of-concept this is enough.

The funny bit is in the bazel argument, what's this about? There is an unexpected issue coming from using bazel provisioned by nixpkgs (which I guess is the most common case for rules_nixpkgs users): internal bazel files like @bazel_tools//tools/cpp:linux_cc_wrapper.sh.tpl get the shebang mangled so that #!/usr/bin/env bash, becomes, say #!/nix/store/5nr4gffsb5l72xpz6d5jbd0skih3r9i2-bash/bin/bash and the problem is how to ensure that those paths ends up in the docker image as well. The workaround here is for the user to optionally provide a nixpkgs_package that should correspond to the exact same bazel being used on dev, CI, etc., so that whatever was used when mangling shebangs is available as a dependency.

As far as I can see, this issue goes beyond rules_nixpkgs, in that a nixpkgs-provisioned bazel is hard to use with remote execution and may require some upstream fix. I think all the possible approaches listed in #180 would be affected by this.